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FOREWORD 



This publication is^one in a series of National Center for Research in Vocational Education 
handbooks on vocationai education evaluation. A primary purpose for this handbook series is to 
offer assistance to persons working to increase the quality of vocational education. Reflected in 
all publications in the handbook series is the intent to advance the theory and practice of evaluation. 
Specifically, it is hoped that the material presented in this handbook will help provoke, stimulate, 
and lead the way toward more efficient use of vocational education evaluation results. 

This handboQk was developed by the Evaluation and Policy Division, the National Center for 
Research in Vocational Education under a contract with the Office of Vocational and Adult 
Education, U.S. Depar^tment of Education. The National Center is particularly indebted to Stephen 
J. Franchak, Project Director; Elizabeth Jen, Graduate Research Associate; and Eliseo Ponce, 
Graduate Research Associate, who had the primary responsibilities for, the preparation of this 
document. Also, recognition and appreciation are extended to Michae^H. Kean, Director, Midwestern 
Regional Office, Educational Testing Service, who contributed an original draft which was used in 
the development of this handbook. Also, significant contributions to the development of this docu- 
ment were made by other members of the National Center's Evaluation and- Policy Division, including: 
N. L McCaslin, Associate Director; F. L. McKinney, Program Director; and William Stevenson, Senior 
Research Specialist. ' 

In addition, the National Center extends its appreciation to the following state and local educati^fii 
personnel who reviewed the draft outline of the handbook: staff members from the Research Coordi- 
nating Unit, Alabama Department of Education; Herb Rand and Mark Headrick, Division of Vocational 
Education, Florida Department of Education; Cheryl A. Rigby and Aaron Gaines, Lively Area Voca- 
tional Center, Leon County, Florida; Rose Mary Bengel, Maryland State Department of Education; 
Florence Sutler, Division of Occupational Education, Planning, Research and Evaluation, New York ^ . 
State Department of Education; Andrea Kelly and Ken^ Lake, South Carolina Department of Education; 
Richard Cothran, Greenville County Schools, Greenville, South Carolina; and Steven Bishopp, Com- 
mission of Vocational Education, State of Washington. 

■ • '\ 

We are also grateful to the eight members of the National Center's Evaluation Technical Advisory 
Panel: George C. Copa, University of Minnesota; Toni Hall, Navar;ro College, Texas; Ruth P. Hughes, 
iowa State University; William R. Morris, Chancellor's Office^ California Community Colleges; Douglas 
Patterson, Alabama State Department of Education; Dolores M. Robinson, Florida State University; 
Robert E. Spillman, Kentucky State Department of Education; and to Tim L. Wentling, University ' 
of Illinois. Credit is also giv6n to the following reviewers of the draft copy: Freda M. Holley, Director 
of the Office of Research and Evaluation, Austin Independent School District; D. Ross Thomson, 
Associate in Educational Research, Bureau of Occupational Education Evaluation, New York State 
Department of Education; William L. Hull, Senior Research Specialist, the National Center for 
Research in Vocational Education, The Ohio State University, 



Finally, a special note of appreciation is extc ided to Sherry White, who had the major 
secretarial responsibilities for this publication, to Kathy Haycook, Kathleen Medley, and Priscilla 
Ciulla for their typing assistance, and to Marilyn Orlando who also provided secretarial assistance 

Robert E. Taylor 
Executive Director * 
National Center for Research 
in Vocational Education 



EXECUTIVE SUMMARY 



The effects of the Education Amendments of 1976 on vocational education evaluation 
practices are beginning to be realized. It appears that more systematic thought is being given to the 
design and conduct of evaluation activities. However, improvement is needed in many evaluation 
areas.and at all educational levels. v 

The complex nature of evaluation, the theory, methods and practices, continue to tax those 
responsible for vocational education evaluation. Moreover, the concerns of Congress and the general ' 
public about the lack of evidence of program impact, and equally important the apparent lack^of 
evaluation use for=decision making ha\/e caused some individuals to give serious thought to the need 
for emphasis on improving the use of eivaluation results'. Recent studies of state vocational education 
evaluation practices reveal that there exists a need to improve formal dissemination and utilization 
efforts. 1 

\ ■ This handbook was designed to help slate and local vocational education evaluators and persons 
using evaluation results. Specifically, the handbook provides ideas for increasing the use of evaluation 
results. We attempted to reduce the current knowledge to the briefest, most practical form in order 
to guide the reader toward developing strategies and procedures for improved use of vocational 
education evaluation results. This handbook is organized to serve as a "ready reference" on the 
subject of utilization of vocational education evaluation results. It is divided into five sections and 
numerous subsections, followed by a glossary and selected and annotated bibliographies. 

Chapter I provides a comprehensive overview of the contents and q summary of-definitions of" 
evaluation utilization. Chapter II presents a^i overview of the progress and trends in evaluatibn- 
. utilization focusing on the federal, state, and local requirements for vocational education evaluation 
Ghap.ter 1 1 1 presents information qn the theory and process of communication, decision making, and 
innovation as these factors may apply to evaluation utilization. This discussion focuses or> these 
factors as comprising a conceptual framework for the use of evaluaton results. Chapter IV concen- 
trates on solutions to various utilization problems encountered by the vocational education personnel 
at all educational levels. An extensive series of approaches recommended for promoting the use of 
evaluative data and information are examined anfi developed. Chapter V describes selected evaluatiojn 
Jata preparation and presentation strategies and techniques for increasing the utilization of evaluation 
results. It.further highlights selected key approaches recommended in Chapter IV. 

This handbook also provides a glossary of terms which we consider to reflect the language of 
evaluation utilization. Many of these terms are adopte'd from the literature dealing with "Change, 
Innovation, Dissemination, and Diffusion." In addition, a selected bibliography arid an annotated 
bibliography, containing detailed summaries of major books and reports conducted focusing on 
evaluation utilization, are presented to stimulate further thought and reading on the subject. 



Where appropriate, checklists are provided to encourage the reader to think through various 
processes or steps in designing or planning fpr the improvement of evaluation utilization. Also, key 
references are incorporated to provide the reader with relevant information about major concepts 
and specific content to make informed decisions about alternative utilization strategies and procedu 

In sum, this handbook presents readers with the kinds of questions they need to ask about the 
evaluation efforts in-order to improve the probabilities of utilization. In particular, it emphasizes 
their need to see the potential for the use of evaluation results for decision making and vocational 
education program improvement. We view this handbook as only a beginning, but in undertaking - 
this effort we join a growing number in the field of education who are identifying significant issues 
bearing on evaluation utilization. 
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CHAPTER I 
INTRODUCTION 

Background 



The primary purpose of this handbook is to help state and local vocational education personnel 
responsible for evaluation increase the utilization of evaluation results. With the rapid growth and 
development of vocation education, the need for evaluation is both urgent and timely. In response to 
this need, federal funding legislation that includes provisions for funding has given increased priority 
to evaluation of program quality. ' 

In the past two decacfes, the practice of evaluation has madeVapid gains, having become more 
systematic and manageable. However, recent studies of evaluation efforts report scant evidence of 
a component to disseminate data or indicate their use. This def iciency clearly hinders attempts to 
achieve recognition and acceptance of evaluation efforts. [ 

A major objective of this handbook is to provide practical guidelines, strategies, and procedures 
that will enable readers to improve their delivery of evaluation findings, and thus increase the 
probabilities of utilization occurring. Because th% dissemination of ^valuation data alone seldom 
fosters change, the handbook focuses on theory and practice directed toward planned change.'" 
Research on planned change for vocational education is relatively new, an^ ,is considered to be in 
its beginning stages (Beuke and Farrar 1979). Therefore^ many of the strategies and procedures 
presented here have been derived from other fields, including information Sciences, agriculture, and 
education in general. ./^ 

^Given that the reader^ understand that the utilization of evaluation's of primary importance; 
and given that they sense the heed to improve and the willingness to improve upon this priority 
and problem of evaluation utilization, those readers need to ask, 'What is this job that needs to be 
, done and what are its objectives and. tasks?" Weiss (1980) states that it is important to understand, 
what evaluation utilization means and to understand what it is expected to be used-what knowledge 
is, and what kinds of knowledge are contributed by research and evaluation. (p. 79). 

In reviewing the book, Using Evaluations: Does Evaluation Make A Difference?, by Alkin et al. 
(1979), Johnson (1980) states that the authors make the point about the varieties of utilization that 
can occur; however, he states that it barely makes a start toward providing handy rules of thumb 
for evaluatiors who wish to be more effective (p. 93). This handbook'attemptsto address that 
change. . \ 

In summary, some vocational education evaluators may find that this handbook can provide 
many insights, strategies, and approaches fo^a wide variety of problem^s on the use of evaluation 
results.^Also, it can serve as a catalog.of avallable.methods and techniques. Thus, it is the intention 
that this handbook be used as a 'Veady reference" or decision making companion for improving ^ 
the use of evaluation results. f\ 



.. .^ Developmerit of the Handbook 

Selected consultants, advisory panel mesnbers, and state and local practitioners responsible for 
vocational education evaluation assisted in tr3 formative and summative stages of this handbook. 
The literature review forms the primary information base for the development of contents. This 
publication draws extensively from the following works in particular. 

Using Evaluations: Does Evaluation Make a Difference? Alkin, Daillak, and White. (1 979) 

"Utilization of Evaluation Information." Braskamp and Brown. New Directions for Program 
Evaluation, (1980) 

Planning for Innovation through Dissemination and Utilization of Knowledge, Havelock. 
(1969) . . 

A Workbook of Checklists to Accompany the Change Agent's Guide to Innovation in 
Education. Havelock. (1973) 

Research and Development Utilization Strategies and Functions: An Analytical Comparison 
of Four Systems. Havelock and Lingwood. (1973) 

/ ' . • 

Utilization-focused Evaluation, Patton. (1978) 

"Incentives for Innovation in the Public Schools!" Pincus. Review of Educational Research 
Vol. 22. No. 1. (1974) 

"Utilization of Evaluation: Toward Comparative Study." Weiss. C. H. Weiss, ed. Evaluating 
Action Programs: Readings in Social Action and Education. (1972) 

Using Social Research in Public Policy Making. Weiss (ed.) (1977) 

Innovations and Organizations, Zaltman, Duncan and Holbek. (1973) 

Organization o? the Handbook 

! ■ . , 

The handbook is divided into five chapters with numerouslsections and subsections. Chapter II 
reviews federal, state, and local requirements for vocational education evaluation and focuses on 
utility standards established by the Joint Committee on Standards for Vocational Education. 
Chapter III highlights theories and processes of communication, decision making, and innovation ' 
that influence utilization of evaluation. Chapter IV identifies relevant problems and discusses a 
variety of approaches recommended to promote utilization, such as identifying an audience, targeting 
a report, dealing with areas of resistance, and establishing credibility. Chapter V further examines 
reporting dissemination strategies and presents. four scenarios illustrating situations in which utiliza- 
tion has occurred. Both Chapters IV and V are directed primarily to persons conducting evaluations on 
state and local levels. Pleaders who wish to become more familiar with the utitizatron research are 
encouraged to review the annotated and selected bibliographies. 

. / 
/ 

Definition{s) of Evaluation Utilization 

What isevaluation utilization? Evaluators need to have better understanding of what evaluation 
utilization means, the types of utilization that can occur, and what kinds of knowledge are contributed 
by evaluation (Weiss 1980, p. 79). 



According to Braskimp and Brown (1980), 

One of the earliest and most commonly used definition states that utilization 
should be measured by the degree to which evaluation has immediate and 
direct influence resulting in specific decisions regarding program allocations 
and changes. The impact of the evaluatbn is obvious, clear, direct and 
reasonable. If change ^nd innovation do not resLjit, it is implied that 
utilization did not occur (p. VIII). 

More recently, the definition of utilisation has taken other forms. "Utilization does not 
necessarily mean that all of the evaluator'^ recommendations are implemented immediately, or that 
all decisions are based directly on the information," Braskamp and Brown explain (p. 91). In fact: 

The evaluation information, for example, may raise larger issues or provide 
policy changes seemingly unrelated to the evaluation issues. There may also 
be a latency period between the end of the evaluation activity and the time 
when decisions about the program ar^e made. In some instances, an evaluation 
may s^rve no role other than a provocative one in which it has no apparent 
. immediate effect but does stimulate intefest, raise new issues, and serve as 
the basis for future evaluation activities [italics addedl (p. 92), 

Smith (1980) also distinguishes between the narrow and broad definitions of evaluation 
utilization. He focuses on the distinction between evaluation use and evaluation impact, indicating 
that one can occur without the other (pp. 24:25), 

Alkin, Daillak, and White (1979) agree that. utilization can. occur anywhere between two points 
on a continuum: one point being any time that anyone uses anything from the evaluation for any 
purpose, the other being when an intended user makes a^ specific decision immediately following the 
receipt of an evaluation report and solely based upon the findings of that report (p. 226), 

From a local education agency (LEA) perspective, Ho!!ey et al. (1979) define utilization 
within the context of an educational program improvement model. They believe that utilization 
is one component of a three-component model for educational improvement which involves: 

1. choosing an important educational question, I 

2. designing and conducting a high-quality study that supplies information about 
that question, and 

3. Utilizing those findings in decision making and action relative to improvement 
in the area in question (p, 2). . ^ 

! 

They aldd that utilization does not occur spontaneously upon the completioii of an evaluation 
study, FurtheK throughout the total process, utilization must have been integral tn the thinking of 
both those conducting the study and those who must eventually use its findings. 

r 

Boruch and Cordray (1980) believe that the absence of any uniform definition^for ''use of 
evaluation results'* underlies some of thsargument about whether they are indeed used (p. 61). 
They offer three broadfunctional definitions: 



. 1. Use of information in making specif ic decisions: _ 

This use may involve modifying. program operation or regulation, developing 
legislation, or constructing specific policy. . ^ v 

2. Use of information to enhance understanding of issues: ^ 
This use encompasses understartding issues, providing context and background for 
policy development, and influencing ideas and attitude about a program. 

3. Use of information to persuade ot/iers or to confirm oner's beliefs: 

The use of information to persuade others, to argue for program changes and levels 
of program support, and other related uses of evaluation are common (pp. 6-1 , 6-2). 

/'■ ■" • 

In summary, the number of evaluaj:fon utilization definitions offered seem to have common 
factors. These common factors appear to be captured in the previously mentioned three.f unctional 
definitions by Boruch and Cordray. It is important that the vocational education evaluators attempt 
to adopt or adapt such a definition(s). Moreover, it is important that they attempt to arrive at some 
definition(s) agreement with i:heir clients, such as administrators, program managers, teachers, etC; 
This is a first step in improving the use of evaluation results. 
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CHAPTER II 

EVALUATION UTILIZATION: PROGRESS AND PROMISE 



^ Background 

The 1 976 Education Amendments reflected the concern of Congress that there was inadequate 
inforrnation on federal, st^te, and local fund usage to realize vocational education policy accom- 
plishments (Martin 1978, p. 1). The passage of this legislation provided much impetus for state and 
local personnel to develop, expand, and redesign their evaluation activities. The enactment of this 
legislation reinforced the need for more effective planning and more sophisticated means, of collect- 
ing and reporting evaluation data. Since then significant strides have been made, but the extent tp 
which all this required activity has been or can be utilized is subject to close;r scrutiny. 

This chapter begins to address the question by assessing the progress that has been made in 
view of federal, state, and local requirements for vocational education evaluation. The utility standards 
establisjied by the Joint Committee on Standards for Education Evaluation are presented with the 
suggestion that readers recall their own evaluation efforts-ejther as producers or as users~and deter- 
mine the extent to which these utility standards were applied. Adherence to these utility standards 
by vocational education evaluators holds promise for improvement in the use of evaluation results. 



Federal Requirements, 
P. L. 94-482, Sections 107, 108, 112 (1) (b) 

The many federal programs instituted to address the human resources problems in the past 
twenty years face a citizen concern for accountability. Changing social and economic conditions ' 
qontinue to support the need for improved decision making and legislative and policy formulations. 
Now, more than ever before, elected representatives must be able to argue from a factual base, some- 
times in light of awesome technicalities (Zweig 1979, p. 2). 

Regulations for the 1976 Education Arnendments focus primarily on the evaluation of program 
quality by requesting data on effectiveness relating to (1) program planning and operational processes, 
(2) student achievement of competencies, (3) student employment success, and (4) reSljIts of addi- 
tional services for special populations., v^i 



State Requirements ' 

State evaluations of vocational education programs have often been described as efforts that 
are conducted on an informal ahd unsystematic basis, dependent-^'upon the initiative of local 
persbnnel:,Ttjis has been supported by evidence of both the 1974 General Accounting Office 
Report and the 1 975 Congressional Hearings on the effectiveness of vocational prbgrams. 



Most state effnrts focus ort program review and follow-up components, with little emphasis 
on the area of studer achievement evaluation. Recent studies identify the need for more clearly 
defined procedures to disseminate evaluation data and promote utilization. It would appear that 
the evaluation requ; - r-ients specified in RL 94-482 call for resources and expertise which - 
state staffs do not have. ' 

In response to the 1976 Education. Amendments, states are shifting from a decentraK: 
evaluation effort in which the local evaluation agencies (LEAs) and institutional staffs desi; 
implement, and analyze evaluations, to a state-controlled system in which the state evaluati. 
agency (SEA) designs standardized evaluation instruments, delineates evaluation procedures 
aggregates and analyzes the data. Although some states will continue to rely on local staffs . 
actual evaluations, tfffe content of the evaluations wfTi largely be determined at the state level 
and Holt 1979). 

This shift to state-based evaluations requires efficient data processing and data analysis capa- 
bilities at the state level. State Management Information Systems (MIS) which can be considered as 
an integral p^rt of an evaluation system are relatively uncommon at the present time, This lack of 
an MIS can contribute to the weakness of the dissemination and utilization component. 



> Local Requirements 

Local evaluation efforts, for the most part, have been initiated because of legislative require- 
ments from both f ' tate and federal levels. A number of exemplary and comprehensive efforts are 
evident in large ck ^iichom districts which possess the expertise and resources to support evaluation. / 
For example, thei^-uHtin Independent School District,' Dallas Independent SchooLDistrict, New York 
City School DistrttiL- Philadelphia School District, Portland School District, and the Milwaukee 
School District have ell demtonstrated; exemplary evaluation efforts. However, each of these school 
districts has identified a need for improvement iri the utilization of evaluation results. 



Evaluation Standards 

The growing concern of Congress and others with evaluators' ability to determine the effect 
of multimillion-dollar programs prompted the formation of the Joint Committee on Standards for 
Educational Evaluation. The committee's work began in the autujnn of 1975 for the purpose of 
providing standards for groups producing and using evaluation reports. Twenty-nine separate standards 
were identified gnd defined. These are categorized under four groups: utility, feasibility, propriety, 
and accuracy. 

. These standards, which apply to the entire range of tasks in an evaluation, are intended for 
persons who commission, conduct, or use the results of evaluations. The tasks are defined as 
contracting, auditing, designing, conducting, analyzing, interpreting, reporting, and using. The focus 
is on standards for evaluating programs, projects:, and materials, rather than institutions, professional 
personnel, or individual students. 

Stufflebeam (1980), chairperson of the Joint Committee,.cites utility as the most important 
evaluation standard. He statesthat the committee's rationale is self-evident: evaluation should not be 
done at all (1 ) if there is no prospect for its being useful to some audience, (2) if it is not feasible to 
conduct it in political, practical, cost-effectiveness terms, and (3) if evaluators cannot demonstrate 
that it will be conducted fairly and ethically. 



Utility Standards 



The utility standards are intended to ensure that an evaluation will serve the practice.; information 
needs of given ^udiences.^Stufflebeam (1980) states that they should not be considered as mechanical 
rules, but rather as guiding principles for evaluators (p. 9). 

• Audience identification] Audiences involved in or affected by tne evaluation should be 
identified, so that their needs can be addressed. 

• Evaluator credibility. The persons conducting the evaluation should be both trustworthy 
and conmpelent tcrperform the evaluation, so.that their findings have maximum credibility 
and achieve acceptan^ce. 

• Information scope and selection. Information collected shQuld be of such scope and 
selected in such ways ks to address pertinent questions about the object of the evaluation 
and be responsive to the needs and interests of specified audiences. \^ 

• Valuational interpretation. The perspective, procedures, and rationale used to interpret 
the findings should be carefully described, so that the bases for value judgments are clear. 

« Report Clarity, The evaluation report should describe the object being evaluated an\its 
context, and the purposes, procedures, and findings of the evaluation so that audiences, 
will readily understand what was done, what information was obtained,. what conclusioihs 
were drawn, and what recommendations were made. 

' • Report dissemination. Evaluation findings should be disseminated to clients and other 
right-to-know audiences who can best use the report information. / 

• Evaluation impact. Evaluation should be planned and conducted in w?ys that encourage 
\ follow-through by members of the audience (Joint Committee on Educational Standards 

for Educational Evaluation 1981, p. 54). 



Evaluator Credibility and Responsibility 

Ultimately, acceptance and utilization of evaluatior findings are contingent on the evaluation 
staff members' ability to get the attention of decision makers and convince them that evaluation 
results can provide a basis for improved decisicr. making and program improvement.. But first, the 
evaluation staff must be able to translate information from complex data systems into a form' under- 
standable and useful to administrators, planners, advisory council members, instructors, and especially 
students and parents. 

^Achieving maximum evaluation utilization also involves drovjding ''understandable" decision- 
making information with implications of alternative interpretations clearly defined. Substantial 
technical skills are required to interpret quantitative analyses fdr the various target audierrces.^ 

Many, evaluators tend to sumniarize their findings in terms ipf single choices; for example, in 
terms of what services are to be delivered to whom by .whom> Yet, as Mahgum et al. (1 979) believe, 
the reality of political forces and conflicting interest necessitates accommodation and compromise. 
Choice is often not between a more effective and a jess effective program but between one of limited 
effectiveness and none at all. Although quantitative data may suggest greater returns for one group 
than another, there may be sound political reasons for preferring the group with the lower cost- 
benefit evidence. And, possibly, the use of qualitative data can provide evidence to support that 
alternative decision (see Spirer 1980). \ 



Many would agree that while evaluators must lean, primarily, in the direction of objective and 
quantifiable bases for decisions, tney must also* accept the necessary compromises of a political 
world-unlessV of course, these compromises conflict unacceptably with the evaluators' moral values. 
KrathwphI (1980) states that all of these bases may be said to revolve around tvvo criteria: "that the 
evaluator has made choices as objectively as possible/ that those choices are perceived as representing 
beneficial prejudice as far as the relevant evaluation audiences are concerned" (p. 25). 

It is worthwhile repeating that evaluative analyses must be tailored to the audiences to whom 
the data and information are to be presented. Presenting evaluative findings to an appropriations 
committee meeting of a state legislature requires one set of techniques;. the techniques appropriate 
in presenting the same findings to a subcommittee of a state advisory council, which may include 
academics and knowledgeable program operators, are quite different. Similarly, multivariate analysis 
results, which may be particularly relevant in explaining the findings from an evaluation of a program 
involving thousands of students in a large metropolitan area, may be totally inappropriate for explain- 
ing the evaluative outcomes for a vocational education program that serves a dozen youth in a rural 
high school. It is not-a difference implied by the knowledge or expertise of the groups addressed; it 
'is simply that complex approaches may not be the most efficient ways of analyzing or presenting 
findings in al! situations. "Evaluation must become a science, but making it useful in a political 
world is an art" (Mangum et al. 1 979). 



Summary 

Some would say the impetus for evaluation activity at the state and local education levels 
stems from the need to comply with federal legislations. Others would say that the individual or 
personal concfern for improving what one is doing is the impetus for evaluation activity. If there is 
an understanding of why the evaluation is done, there is an equal probability for why the evaluation 
result is or is not used. Boruch ,and.Cordray (1.980) state that, in principle, most evaluations are 
carried out to answer questions dealing with: ^ 

(1 ) who is served by the program and who needs the seryices 

(2) a description of the services— effectiveness of delivery and costs 

(3) effects of services oh recipients, and 

(4) cost benefits of alternatives (p. 2-1). 

As difficult as it is to accurately definathe why of evaluation, it is equally difficult to accurately 
def^ine the use of evaluai^ion of results. However, as discussed in the. preceding sections? the definition(s) 
problem, and the extent to which the vocational educational evaluator operationally definesiuse in 
collaboration with his client, determine in large measure the extentrof evaluation utilization. 

\StLidies undertaken to address use have shown varying degrees of impact. But in most instances 
evidence of certain factors involved in the evaluation process seem to increase the probability of 
evaluation utilization. '\ 

Obviously serious thought and effort must take place. In part, the advent of the Joint Committee 
on Standards for Educational Evaluation (1981) proposed "utility standards" holds promise for / 
improvement in the use of evaluation results. / 



CHAPTER III ' 

COMMUNICATION, DECISION MAKING, INNOVATION: 
EFFECTS ON EVALUATION UTILIZATION 

Background 

This chapter presents information on the theory and process of communication, decision 
making,.and innovation as these factors may/apply to evaluation utilization. No intent is made to 
present an exhaustive or all-inclusive review/and synthesis and analysis of the literature or practice. 
The intent, rather^is to highlight conceptS/felevant to the improvement of evaluation utilization. 
The references cited in the text and bibliographies of the handbook provide additional sources for 
a more detailed^examination of those factors. The reader \s encouraged to refer to those sources 
for further study. / . ^ . 

Specifically, this chapter is intended to increase your awareness of: 

• how communication, decision making, and innovation work in relationship 
to evaluation utilization; 

• what effects communication, decision making, and innovation have on the 
evaluation utilization process; 

• what effects communication^, decision making, and Innovation have oathe 
use of evaluation results. 

" > ■ ' / ' ■ ' ■ ' " ■ ■ ' 

This chapter also presents a nujuber of utilization ideas. These are intended to focus on some 

of the many factors whichVelate to the evaluation utilization effort. In most instances, these 

utiiizatipn ideas complement the rpateriarin other chapters of the handbo^ok. The reader is encouraged 

to pause and reflect on^these ideas as they relate to evaluation utilization behavior. 

. . ■ ■ . / ■ ■ ' ' - ■ ■ - ■' ■ 

Communication , 

Communication is frequently defined as a process involving a sender, a message, and ^ receiver. 
Effective communication is defined as being a two-way communicStibn with a fourth element of ■ 
feedback. Diekman (1979) makes the. distinction between communications per se and effective 
communications by stating that qommunication is merely the sending and receivina of both verbal 
and non-verbal messages, whereas.effective communications is productive, helping, and leading 
(p. 12). 

" - . ' . ./> " 

The spoken or written word, aside from numbers and formulas, is by nature inexact and 
flexible. Chase (1954) views every language as a complex system, with its unique noises and patterns, 
which addresses three main functions: ' 

1. to communicate with oneself or, as we say, think ^ 

2. to communicate with jother persons 

3. to mold one's whole outlook on life (p. 101) 



ERIC 



Failure to achieve effective communication is usually the result of misunderstandings and 
. misinterpretations between individuals and groups. Chase identifies twelve kinds of communication, 
failures that provide important ideas for improving the utilization of evaluation results. These V 
include the following: 

• Confusing words vi/ith things 

• Confusing facts with inferences with value judgments 

• Failure to allow for cultural difference 

• Failure to appraise the other person's background and point of view 
» . F«?ilure to appriase motives 

• Failure to assemble the main facts before passing judgment 

• Failure to check abstract terms with concrete events 

• Failure to listen ■ / ' 

• Gobbledygook-prolixity and obscurity; using ten words where one would suffice, 
or drowning meaning in polysyllables or technical terms 

• Overgeneralization 

• Spurious identification: Newspaper guilt-by-verbal association' 

• Wholesale application of two-valued logic; black or white, no gray (p. 11-12) 

tvaluators should be alert to the possibilities of communication failure as they develop strategies 
for the collectiQh and publication of evaluation data and information. In sum, a key ingredient is the 
continual interaction between the vocational education evaluator and the user. The interaction is 
defined as communication (Holley et al. 1979) or it can take the form of collective bargaifiing (Krath- 
wohl 1980). In the words of Confucius: 

"If language is not correct, then what is said is not what is meant; if what is ^ 
said is not what is meant, then what ought to be done remains undone." 



Decision Making 

It is commonly believed thax the evaluator's understanding of the decision making process 
relating to the project being evaluated has a significant influence on the extent to which the evaluation 
information is utilized. The decision-making process can be viewed in light of the scientific method 
(Dewey 1940), problem solving (Ackoff 1979), systems approach, orsystems analysis (Churchman 
1955, Arnold etal. 19.68). 

A beginning point in understanding the decision-making process is to define "decision'\ " / / 
Souder (1980) believes that a.precise definition is very elusive, but offers the following: ^ / 

A decision is a very personal thing that occurs internally with each individual. y 
Judgment and intuition. are involved, as are sentiments and individual value- >-' / 
"systems. We never really get to see a decision. We only see its manifestations, 
and effects. We have observed only the elements of a decision process: Elements / 
equal behavior patterns, the analytical procedures and the sequence of logic used / 
In niaking a decision, (p. 12). . ! 

The decision-making process involves five stages: (1) identification and proper definition of. the 
problem, (2) specification of alternative courses of action, (3) ianalysis of the advantages and dis- 
advantages associated with each course of action, (4) the identification of the best solution or the 
best alternatives, and (5) the follow-up and appraisal of the effectiveness of the decision. Figure 1 
portrays one model of a decision-making process. 



Figure 1 
A Decision Making Process 
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•Arnold, W. M. Vocational Technical and Continuing Education in Pennsylvania: A Systems Approach to State^Local 
Planning. Harrisburg, PA: Department of Publlc.lnstruction and Pennsylvania State Board of Education, 1969, p. 209. 
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According to Stuff lebeam et al. (1972), evaluation is the process of provicJing information upon 
which decisions can be made. It follows that in order for evaluators to serve decision makers, they 
must know first: , ' 

• who the decision makers are; 

• what decision questions they must answer; 

• what decision alternatives are to be considered; 

• what cirteria are to be used in judging alternatives; 

• projected timing of the stepls in the decision process (p. 49). / 

,,/''■ 

The Phi Delta jKappa National Study Commission on Evaluation, chaired by Stuff lebeam (1971) 
classified all decisions in terms of whether they pertained to ends, means, intentions, or actualities. 
Figure 2 presents a/conceptual base for the types of decisions which evaluations can influence, (n this 
framework, all edubational decisions are classified according to (1) intended ends (planning decisions), 
(2) intended meani (structuring decisions), (3) actual means (implementing decisions), or (4) actual 
ends (recycling decisions). 

In the past, /nost evaluation efforts in vocational education have concentrated on the decisions 
as presented in the lower right-hand cell of the matrix presented in Figure 2 (process evaluation). 
Some statewideyevaluation systems, although containing both process and product evaluation, appear 
to focus more pn the process compohent. Ohio's statewide evaluation system (PRIDE), Illinois' Three 
Phase System for Statewide Evaluation, and California's Community College Occupational Programs 
Evaluation System (COPES), are offered as prime examples. Much less attention has been. given to the 
other three^fells of the matrix-particularly to the upper right-hand cell (product evaluation).. However, 
recent studies of state evaluation practices reveal increased attention being given efforts toward product 
evaluation. ^ ^ . 

Stufflebeam's focus on the importance decision making has in evaluation is largely based on the 
findings of Braybrooke and Lindbloom in their classic book, A Strategy of Decision: Policy Evaluation 
as a Social Process (1963). "In presenting a strategy of evaluation we shall in fact have to outline a ' 
problem-solving or decision making strategy," they state, adding, "We have been setting the stage for 
a strategy of evaluation that is inseparable from a general strategy for decision-making" (p. 57). 

An awareness of the close tie-in evaluation and decision making strotegies should enable evaluators 
to increase the prpbability of utilization occurring. , " " 

It is incumbent upon the evaluator to identify and define, possibly through some systerhs 
analysis techniques, the decision-making process within the organization. An excellent model for . 
this process is offered in Data for Decisions by H. M. Brickeil et al. (1974). They conducted a study 
designed to identify the many decision makers associated with a federally-funded project, to estimate 
their relative importance, to isolate the decisions th^y must make, to determine the information needed 
to answer the questions generated by the defcision situations, to determine how they would rank \- 
assorted evaluationVeports as to probable usefuln^is, and to ascertain' in what medium and'at what 
length and at what time they would like to have the evaluation reports. This should be donq prior 
.tp_lheilesign^itiie_ev^ of evaluation objectives 

and questions. The chances of evaluation utilization occurring are largely determined by the extent 
to which evaluator can define the decision-makmg process. Checklist 1 presented three questions for 
diagnosis of the decision maker's problem.. 

Upon defining the process, it is recommended that the evaluator assume the role of the negotiator' 
(Krathwohl 1979). In the scheme of collective bargaining the evaluator must meet with the key actors 
in the decision-making process and engage.in dialogue to determine what evaluation objectives and 
questions should be identified (p. 29)./ . / . 



FIGURE 2 
Typology of Decisions 
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CHECKLIST 1 
tMagnosis of Decision-Makers' Problem* 



How does the deicisjon-maker initially define the problem? 




3. Are there important differences between and "b"? 
■■ No. / . \ • 
^ " ' Yes. What are the differences? 



Adapted from A Workbook of Checklists to Accompany the Change Aunt's Guide to Innovation in 
Education by R.'-C. Havelock. Institute for Social Research, University Michigan, January 1973. 



However, identifying the decision makers and, more importantly, the decision-making process 
within an organization at a state and local level is a difficult task. GiJSkin (1980) states: 

' i ) 

"Decision makers and those who create knowledge are often not able to relate 
to each other, and when they do, each has difficulty understanding the world 
in which the other lives and operates. For while information is an essential 
resource for decision makers, the manner in which it is converted into policy 
is based as much or more on interpersonal, organizational, and psychological 
factors than on the actual information itself" (p. 14). 



- innovation 

Many critics consider the literature on innovation to be inadequate from a scientific viewpoint. 
In their work, Beuke and Farrar (1979) cite many references indicating the lack of methodological 
sophistication and a poor conceptualization of the change process. Fullan (1978) is also critical of 
the literature, but is optimistic that much of the work is prerequisite to the formulation of an 
adequate theory which might guide practice. 

Probably the most popular and most frequently cited work in the field of education innovation 
is Havelock. Havelock's (1969) four models of education innovation can provide the evaluator with 
awareness ideas for improving the prob^biJijy of evaluation utilization. The four models are 
(1)^ Research, Development, and I^jfJ^sion IW)^I, (2) Problem-Solving Model, (3) the iSocial Inter- < 
action Model, and (4) Linkage Model. 

The Research, Development, and Diffusion Model is based on the'assumption that agencies 
outside the system of potential users should develop the innovation products and present the 
completed package to users, i.e. school administrators, teachers, and so forth. A major criticism of 
the model is that little attention is focused on the user system. ^ 

The Problem-Solving Model emphasizes the importance of the user as the primary source of 
interest in and awareness of ^.the peed for change. The potential users, e.g. teachers, are involved in 
the planning and analysis of needs and identification of problem(s) before the innovation is selected. 

The Social Interaction Model places emphasis on the externaLresource as the initiator of the 
change strategy. The model stresses the influence of opinion leaders. The success of the innovation 
is based on personal contacts between resources and key users in the user system. 

The Linkage Model combines elements of the Problem-Solving and Social Interaction Models. 
Both the user and resource systems unite in close reciprocal cornmuriication to diagnose problems 
and search for solutions. 

• »^ 

Beuke and Farrar (1979) point out that these four models of educational change are based on 
a rationalistic bias of human behavior which assumes educators have clear goals, posseiss techno- 
logical sophistication, seek better practices, are able to identify superior products and processes, 
and are eager and able to adopt proven innovations. However, little evidence exists to support 
those assumptions. * 
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Strategies for Innovation Implementation 



Research and practice reveals that individuals and agencies vary widely in their willingness to 
adopt new programs or practices (Hull and Bina 1977). A number of strategies suggest ideas for 
increasing the probability of evaluation utilization. Zaitman, Florio, and Sikorski (1977) defined 
four strategies: (1) power strategy, (2) re-educative strategy, (3) marketing technology strategy, 
and (4) manipulative strategy. 

The power strategy is generally used when the change agent feels voluntary action by the 
intended adopter or user is unlikely, the success of a power strategy usually rests on the extent to 
which the sources of power are really valued or compelling. ' 



The re-educative strategy rests on two basic assumptions: one, that people are guided by 
reason, and two, that people will follow their rational best interest vvhen it is revealed to them. 
This strategy generally requires well-defined, clearly agreed-upon goals. 

. The marketing technology strategy involves identification of marketing opportunities, 
coordination of the delivery of new products, and feeding market information back into the 
production activity. This strategy has a prinriary focus on the users' needs. 

The manipulative strategy involves persuasion through the presentation of biased information 
and use of facilitation methods that cause change to be implemented effectively. This strategy, 
more than any other, requires an in-depth knowledge of the target systems, change agent, and the 
change itself. 



This chapter presented information on the theories and processes of communication, decision 
making,, and innovation that should be considered in. planning to increase the probability of 
evaluation utilization occurring. The fundamental importance of clear . and concise communication, 
both oral and written, was also emphasized, along with the importance of decision making as the ' 
key factor in developing an evaluation strategy. Further, the important role of/the decision-making 
processes in innovation was defined/ in view of the fact that decision makers are often faced with 
choices to innovate or not, to select different innovations, or to use different methods of imple- 
mentation (Zaitman, Duncan, and Holbek 1973, p. 53), 

Within a state or local educatfon environment vocational education evaluators should define 
or identify those factors associated with the communication process, the decision-making process, 
and innovation process. Moreover, it can be hypothesized that there exists within each unique 
environmental setting at either the state or local education agency level a set of utilization factors. 
And that these factors can be considered as barriers or faciiitators^ for evalu&Lion utilization. With 
this knowledge available at the design stage of an evaluation an:J the use of this information in 
designing the evaluation plan, the probability for maximization of evaluation utilization can be 
enhanced. Therefore, the analysis of the environment context is considered to be extremely 
impprtant, and vocational education evaluators must have an understanding of the communication, 
decision making, and innovation process within their respective state and local^ education agency. 



Summary 
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CHAPTER V 

PROMOTING THE UTILIZATION OF EVALUATION DATA 



Background 

The growing tendency for local education agencies 'to provide for their own research and 
evaluation needs is marked by the hundreds of school-based research and evaluation off ic2s already 
in existence and the sizeable number beijicj planned. With the growth of in-house evaluation services, 
common problems are more readily observed. 

Perhaps the most serious initial problem is one of survival as a viable organizational unit. 
Survival of LE A-base ;! offices of research and evaluation may well be directly related to their ^ 
ability to meet their clients' information needs and to communicate data to them in a readily 
understandable and useful format— a far from easy accomplishment, however. 

Failure to communicate evaluation findings and to indicate their use represents, perhaps, the 
most serious deficiency of attempts by the profession to achieve recognition of the utility of their 
"products." 

The major purpose of this'chapter is to concentrate on solutions, not problems. As-such, an \ 
extensive series of approaches recommended for promoting utilization of evaluation data will be 
developed and examined. Many of the approaches reinforce the need to adhere to evaluation 
standards/ such as those mentioned in Chapter 11. ' ^ 

Underlying these approaches is the assumption that the factors affecting the utilization of 
evaluation in vocational education are basically the same as for other program areas. Certainly, 
voqational education evaluation should be treated according to the special needs imposed by the 
parameters of the program itself. This is not to say, however, that the factors influencing utilization 
of vocational educatipn evaluation are different from those in other disciplines. It would be par- 
ticularly unfortunate, because of tradition or governmental mandate, to view the utilization of 
vocational education evaluations as an automatic process. In a recent report examining state practices 
and procedures for assessing vocational education programs in light of the 1976 legislation Smith 
and Holt ( 1 979) conclude that: . ' 

Lacking from current state evaluation procedures is a formal dissemination and 
utilization mechanism which specifies how evaluation data are used for program 
planning and improvement. In fact, the use made of evaluation results is the 
least documented element of state practices. This may be a result of states' 
attitude to or perception of evaluation in general and their inexperience in 
using evaluation data in particular (p. 30), 
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Utilization Problems and Factors 



In her presentation at the second annual Johns Hopkins University National Symposium on 
Education Researco, "OThou ffringest The Tidings to Lions: Reporting the Findings of Education 
Evaluations/' Lois-cllen Datta examined three categories of sources in an attempt to demonstrate 
that little attention is paid to ,"the evaluation end game." Having reviewed over a hundred, books 
written on evaluation during the past decade, Datta (1979) selected three recent works for comment: 
Handbook of Vocational Education Evaluation by Abramson et al., ^979: The 1979 Evaluation 
Studies Review Annual; ar\6 Evaluating Instructional Programs by Tuckman. In all three, she observed, 
'The authors devote considerable space to clarifying objectives for the programs to be evaluated and 
the purposes of evaluation. Research design receives as many chapters, and measurement, the rest." 
What is most critical, Datta found, is that communication and utilization of evaluation findings are 
rarely included as topics. 

Datta's overview of selected research reports and a detailed examination of several major case^ 
studies including Data for Decision Makers by Brickell, 'Aslanian and Spak (1974), and Using Evalu- 
ations: Does Evaluation Make a Difference? by Alkin, Daillak and White (1979^, produced somewhat/ 
richer results. These were classified under four broad categories: (1) the decision maker /decision 
process, (2) the program social context of the program, (3),the nature of the evaluator, and (4) the 
evaluation process/evaluation report (Alkin, Daillak, and White \979, p. 19). 

Finally, in an attempt to review "the state of the practice," Data (1979) examined several recent 
studies: Evaluation of Vocational Education: Roles, Responsibilities, and Responses of State and 
Federal Agencies by the National Center for Research in Vocational Education, 1979; "The Effects 
of the 1976 Legislation upon State Evaluation," put on by The Vocational Education Study: The 
Interim Report, Henry David, ed., 1 980; State of the Art Review of Vocational education Evaluation: 
State Evaluation Procedures and Practices, by Smith and Holt; 1979; and A Study of State and Local 
Compliance and Evaluation Practices: First Draft of Final Report, by Beuke et al., 1980. This 
investigation led Datta (1979) to conclude that the most useful information remains to be found 
out in the field-with the local education agencies-where it is actually occurring, not in publications. 
A need for further research in this area is obviously indicated. / 

There'are increasing numbers of school-based offices of research and evaluation producing many 
thousands of research and evaluation reports. The quantity of locally produced reports can easily be 
verified by examining the annual additions to the various ERIC clearinghouses. Many of the large 
city school systems (those with enrollments in excess of 45,000 pupils) participate in a group called 
the Directors of Research and Evaluation (DRE),, which also lists a^elected sampling of evaluation 
reports on an annual basis. Over 200 reports were entered in the 1979 American Educational Research 
Association Division H (School Evaluation and Program Development) Evaluation Awards Competi- 
tion. Based upon the scant number entered in the category relating to evaluation impact upon board 
policy development and decisions, however, it is not unreasonable to assume that little is known 
about the impact of these reports on school district policy. ' 

Why is this so? Why do certain research and evaluation reports affect policy and others do not? 
Is impact related to the eight factors identified by Alkin, Daillak, and White (1979): (1) pre-existing 
evaluation boundaries, (2) orientation of users, (3) evaluators' approach, (4) evaluator credibility, 
(5) organizational factors, (6) extra-organizational factors, (7) information content and reporting, 
and (8) administrators' styles? Or are the recommendations made by Brickell, Aslanian and Spak 
(1974): (1) brevity, (2). isolation of technical material, (3) timeliness, (4) nontechnical language, 
and (5) an illustrated executive summary of the prihiary factors relating to utilization impact. 

, . O- 
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Still sn'other approach to viewing the factors related to utilization was developed by Holley 
(1979)cShe grouped the factors into six categories, five of which deal with characteristics of (1) the 
program being evaluated, (2) the user, (3). the context (organization), (4) the findings, and (5) the 
evaluator. The sixth category relates to the resources available to support dissemination and utiliza- 
tion of the evaluation. ^ 

As discussed in Chapter III, the importance of defining and understanding the environmentah 
context of the state or local education agency within which the evaluation takes place cannot be 
overstated. The communication process, the decision-making process, and the innovation process 
which make up the environmental context, and those factors associated with those processes, are 
extremely important for improving the utilization of evaluation results. The factors mentioned above 
are a part of this environmental context, and it is incumbent upon vocational education evaluators 
to understand this context and make use of this information in the evaluation design. If this is done 
the probability of increasing usefulness of the evaluation results will be enhanced. 

Instead of isolating additional factors or creating a philosophical and theoretical" construct on 
which to hahg such factors or variables, the following sections will attempt to identify a series of 
specific approaches designed to promote the ,utilization of evaluation data. The approaches are 
presented as potential solutions to the probleni. Checklist 2 is provided for readers to assess their 
own utilization roles. ' 



Utilization Approaches 

Identification of Clients 

It is important. to single out the specific clients and client groups. This identification should be 
based on those most closely related to the evaluation and most directly affected by any policy 
decisions likely to be developed as a result of the e valuation. However, others not closely related 
should be .identified. The expectations of the clients, as well as the type of information and services 
that they require, are vital considerations in focusing the reseafch and evaluation process. The clients 
must first be identified before their expectations can be addresse,d. 

A recent National Institute of Education (NIE) study identified thirty-nine different clients 
and client groups for evaluation studies. These were divided into two categories: direct clientele and 
indirect clientele. Direct clientele includes those individuals or groups served as the result of organi- 
zational intent, hierarchial relationships, and fiscal support, e.g., the superintendent, project directors, 
principals, or' teachers. Generally speaking, these persons have a basic right to receive services. 

Indirect clientele includes individuals or groups having an even greater need for services than 
certain of the direct clientele. They usually request information rather than demand it. Included in 
this group are members of the community, media, and student body. The requests^of indirect clientele 
may be forwarded through other agencies or individuals who are themselves direct clientele. 

Early identification of the evaluation's clients is critical and should take place during the 
planning phase. Doing so is more likely to produce a.better and more acceptable product. 

Checklist 3 enumerates certain direct and indirect user groups that vocatipnaT education 
evaluators should consider in developing evaluation plans. 
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CHECKLIST 2 
Preliminary Self-Assessment and 



1. The primary utilization role you, the evaluator, follow: 



Needs 

Good Adequate Improvement 



Training and experience for role 

Formal status in this role 

Informal social support and 
recognition in this role 

Feeling of persona! competence 
in this role 



What other utilization roles do 
you adopt at times? 

Overall feeling of competence 
in this role 



Adapted from A Workbook of Checklists to Accompany the Changi Agent's Guide to Innovation in Education 
by R. C. Havelock. Institute for Social Research, University of Michigan, January 1973. 
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Encouraging Active Client Participation in Implementation 

It is importaht to involve as many groups as possible who will ultimately be affected by the 
evaluation. After all, it is their program. This irivolvernent should be genuine, not mere "window 
dressing/' Though this' procedure may lengthen the period of the eivoluation, it will pay handsome 
dividends in the long run. 

To do so, of course, assumes prior identification of the evaluation's clients, whose selectioru-^ 
requires special emphasis. Clients should participate in a broad variety of evaluation activities: 

• Stating program objectives 

• ^Stating questions to be answered 

® .Determining timing of information. 

• Designing the reporting format 

• Briefings (interior feedback) 

• Presenting tentative data 

• Reviewing final draft of the report 

Guaranteeing Technical Accuracy 

The technical aspects of the evaluation-research methodology, measurement techniques, and 
reporting— must be above reproach. Wherever possible, the evaluation should improve on previous 
major initiatives in the area, and to this end, the use of "multiple measures will greatly enhance the 
study's technical accuracy. 

A single crack in the purity of the methodology and analysis will definitely discredit the 
entire evaluation effort. It is unnecessary, however, to become.trapped into a classical (experi- 
mental/control, preVpostresearch) design. The state of the art in evaluation methodology is now 
adequate to provide a tecl;inlcally rigorous comprehensive aslsessment and still focus on specific 
program objectives. It is also unnecessary and generally inappropriate to select ultrasophisticated 
statis^cs if simpler forms of data analysis will provide the same quality of information. 

For more information on accuracy standards, see the Joint Gomrhittee on Standards for 
Educational Evaluation (1981, pp. 97-141). 



Planning the Sequence and Determ:;iing 
the Frequency of Dissemination 

Dissemination of results is, of course, an integral part of the evaluation process, and someone 
must be charged with the responsibility for overseeing the distribution of reports and scheduling 
presentations. This dissemination responsibility should be determined in advance. It is the evaluator 
who will usually disseminate the information to the primary clients. Broader authorisation for 
dissemination may be useful. Sometimes an agreement for funding of dissemination can be worked 
out prior to the start of the evaluation process. 

Planning the sequence, as used here, has a double meaning: (1) serving primary clients first, 
then secondary clients, (2) establishing the sequence or order of the presentation (e.g., project' 
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CHECKLISTS 

Defining and Ranking 
Direct and Indirect User Groups 



\ 

Group Types 


Rank 
Major Order 


Rank 
Minor Order 


Not.; 

Applicable 


Agencies 








Comprehensive Employment and Training 
Act (CETA) Prime Sponsors 


■ □ 


□ 




Community-Based. Organizations 




■ □ 


□ 


County or Intermediate School Units 


□ 




□ 


Local Advisory Committees 




□ 


□ 


Local Boards of Education 


□ 


□ 


□ 


Local Business and Industry 




□ 




Local Postsecondary Institutions 


1 I- 


1 ! 

Lj 


I 1 

□ 


National Occupational Information 
Coordinating Committee (NOI CO . 


□ . 


□ 


■ □ 


State Occupational Information 
Coordinating Committee (SOICC) 


n 


□ 


□ 


State Denartment of Eduratinn ' 


□ 




□ 


State Departments, such as . i 
Labor and Commerce 


□ 


□ 


□ 


U.S. Departrhent of Education 


□ 


□ 


□ 


U.S. Departments, such as 
Labor and Commerce 


□ 




□ 


Persons .. 








Administrators 


□ 


□ 


□ 


Current Students 


. □ ■ 




□ . ■ 


Former Students 


n _ 


□ ■ ■ 


□ 


Student Personnel Services Staff such as 
Counselors and Job Placement Specialists 


□ 


. □ 


□ 


Teachers 


□ 


□ 


□ 


Supervisors 


□ • 


□ 


□ ' - 
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description, evaluation approach, results, implications). This sequence can be expinded, depending 
upon the types of audiences to whom the presentation is made. It should be remembered, however, 
that the findings must be presented first. . . " 

Finally, frequency should be considered. Single distribution or several waves of dissemination 
are possible choices. Multiplepresentations (utilizing different forms of the report) are often 
valuable. " 



Keying the Results to ' - 

Timing the Decisions . 

Timing is all-important if evaluation results are to have a genuine impact on policy. For effective 
timing, it may be useful to attempt to determine the type of decision expected as a result of the 
study. , 

The'evaluator should determine the decision cycle in advance. The types of decisions and 'the 
key time requirements should also be considered. Allowing sufficient" lead time for consideration 
of the results is important, and meeting deadlines is imperative! 

Including an Assessment of Those 
Objectives, Important to Clients 

A major aspect of the program planning process is the refinement of program objectives, a 
process which should be carried out before the program begins, by both che evaluator and program 
' personnel. In this way, the evaluator participates in the development of the program, and carries 
out activities that anticipate the evaluation. The evaluator should pay particular attention to the 
objectives from the point of view of their informational value to the decision maker/primary client. 

The number of objectives selected should be limited. The evaluator can avoid making a simple ' 
project seem too complex by eliminating objectives which may be interesting from an academic 
standpoint, but which are of little value to the client. It is always wise to emphasize the most 
important objective when reporting results, regardless of whether the results are positive or negative. 

Briefing Decision Makers and 
Policymakers in Advance 

, Providing advance briefings to key decision makers and policymakers is of tremendous value. 
Such briefings should include a nontechnical overview of the specific results of the evaluation, the 
relationship of those results to the needs of the system, and the specific implications for policy 
which seem to evolve from the study. 

Because evaluation results, no matter how simplified, tend to be threatening (clients often 
dislike statistics), an advance briefing may prove especially useful in promoting understanding, which 
in turn leads to acceptance and utilization. As an analogy, consider the British cricket players sitting 
in Yankee Stadium watching their first baseball game with no prior knowledge of how the game is 
played. They will naturally rely upon thieir knowledge of the rules of cricket, and in a very short 
time become thoroughly confused. If, however, the purpose of the game, its rules, and its system of 
scoring are explained to them in advance, they will be more likely to understand the game and may 
actually accept and enjoy it. z 

' ■ / • • ■ ' 
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The evaluator should use real examples from th.e project itself in' briefing the policymakers. An 
attempt should be made to stimulate discussion of possible decision/policy alternatives. Simulation* 
of "what If" decisions as well as policy alternatives (e.g., go,.no-go, refocus) may also be useful. 

Dealing with Areas of Resistance 

Change, by nature, is threatening and creates resistance, a fact documented by many studies. 
Evaluation data are likely to result in one of three possible decisions: go (continue), no-go (stop) 
and modify. Two of these three decisions imply change, and even if no change is recommended 
the very anticipation of it may cause some resistance. 

The ability of the evaluation team to anticipate the potential problems created by their work 
will go a long yyay. toward reducing resistance. Dealing with such points of resistance before they 
surface will, in the long run, forestall disruptions and increase the changes of translating the evalu- 
ation results into practice. 



Using an Ombudsperson / 

: Designating a member of the evaluation team as an ombudsperson and a troubleshooter during 
the iniplementation pf thestudy will help reduce both resistance and misunderstanding Such an 
individual should initiate dialogue with evaluation clients before problems occur and should work 
with them prior to, during, and after evaluation implementation. 7 

The role of ombudsperson is not common in vocational education; hence, it must be explained 
to potential users. This person's role is to- minimize disruption or problems created by the evaluation 
effort (or misunderstandings about it), while making the evaluat/on as unobtrusive as possible. 

This role may hot be that of a true ombudsperson in the sense that we know the word- 
however, this individual acts as an advocate for one group (the evaluator or evaluation team) with 
a view toward serving the needs of another group, the clients. ''Having a person in this role ensures 
the ultimate neutrality of the evaluation team in terms of the/project's outcomes. 

Playing the Entrepreneur's Role 

Another factor to be considered as an important correlate of evaluation utilization is the 
entrepreneurial role, a role played by either the individual responsible for producing the evaluative 
information or the decision maker ultimately responsible for translating it into policy and using it. 
If the policymaker exhibits entrepreneurial behavior, the evaluator's role is considerably simplified. 
Since this is not typically the case, however, it is the potential impact of the researcher's entrepre- 
neurial acumen which will be considered here. / 

Within the context of economic development, the entrepreneur is an individual who applies 
a new combination of resources and technology to productive activity in order to effect change. 
The change which results, however, is rarely only economic, but is social as well; for economic 
change does not occur vyjthout social ramifications. There is a great deal of similarity between the 
roles and characteristics of the entrepreneur and the change agent. Going by either classification, 
such an individual has been the key figure in unlocking the doors of development in many of the 
world's progressive nations. 
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The entrepreneurial role would seem to be not only : nriate. but extremely useful to assur 
in working for acceptance of evaluative information, and ulllmo^w.y ;or its utilization. In playing 
the role of the entrepreneur, the evaluator cannot assume a neutral posture but miist be, in fact a 
strong advocate of the utilization of the information that has been developed. From the'beginni'ng, 
the task must focus on translation of the findings into policy. It is'important to rememljer that 
advocacy of the utilization of eva uation results need not compromise objectivity in the conduct 
of the evaluation. 



Using an Evaluation Audit 

The use of an evaluation audit provides an "outside" second opinion. This practice allows the 
evaluation to be audited as a company's books are audited by an accounting firm providing a fiscal 
evaluation of the corporate program. The audit should include several facets of the evaluation: (1) 
methodology, (2) data analysis, (3) conclusions, and (4) implementation, in a way, the audit represents 
an evaluation of the evaluation (metaevaluation). 

The evaluation audit may be viewed asa validity and reliability check, offering an additional 
level of confidence to the decision maker. This process also serves as a type of insurance-actually a 
type of fail-safe mechanism-for the decision maker in the event that-the evaluator should somehow 
fail. 

The evaluation audit should be coordinated with the program manager and evaluators and 
should not be viewed as competitive by the evaluator, but simply as a review and a second opinion. 

For further information on the evaluation audit the reader is encouraged'to review: Improving 
the Accountability of Career Education Programs: Evaluation Gui<^i^nes and Checklists by Adams 
and Walker (1979, pp. 28-30), and Evaluation Guidelines and Practices for State Advisory Councils 
by Stephens (1980, pp. 13-36). 



Establishing the Credentials of the Evaluator 

Ensuring credentials should be attended to in advance, whether an evaluation is being conducted 
by an in-house or outside team. Credentials and back-up support, once established, should result in 
confidence in the evaluation team^^embers. Confidence in the evaluator is essential to confidence in 
the evaluation; likewise, confidence in evaluation results is linked to ultimate utilization. 

There are a number of means of establishing credentials of the evaluator, including, for example, 
previous evaluation work, training, and the recbm^nendations of other "clients." Each time an 
evaluation is completed and the results are satisfactorily utilized, the eval^ator's credentials become 
more firmly established. 



Using Key Power Sources Appropriately 

The povyer behind an organization nrwy be different from the primary clientele of the evaluation. 
Depending on the environmehtal context of the evaluation, key community powt^ brokers, having 
little to do with the particular project, may be vita! to the ultimate utilization of an evaluation. 
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These power sources may ask questions such as whether the evaluation. results were used, how, 
why, or why not. Such inquiries may be made in either an official capacity by, for instance, a' 
member of the chamber of commerce- education (Committee, or in an unofficial capacity by the 
president of a major industry. Such individuals are often professionals or business persons who are 
accustomed to data-based decision, making. If approached properly, they will be supportive of the 
concept of evaluafticn and the utilization of its results. 



Working with Media Representatives 

A good place to begin in estabHshihg relations with the media is to conduct a media seminar. 
Included in such a seminar may be representatives from television, radio, and newspapers, with the 
scfjool district's director of information services or public relations. The project director may be 
involved in the initial oriefing, but usually is not. The evaluator and the head of .the evaluation office 
^ are, of course,, pivotal in such a briefing. Two media seminars are recommended, one to set the stage 
V before the results are released, and another at the time the evaluation is made public. 

/ Personal contacts with the media also play a key role. One shquld always answer all questions 
but should avoid releasing information to one source and not another; If evaluators cannot answer 
a question, they. should say so, since media persons are likely to recognize bluffing. To "get technical" 
will usually backfire. 

Media representatives, simply by presenting the facts to the public, can be powerful influences 
in providing for the utilization of evaluation results. 



Presenting Evaluation as Part of the 
Management Support System 

Evaluation may be thought of as the process of providing information to be used in decision 
making. These decisions may be classified in four categories (Stuff lebeam 1972): (1) planning 
decisions that determine objectives, (2) structuring decisions that design procedures, (3) irpplementing 
decisions that utilize, control, and refine procedures, and (4) recycling decisions that judge attain- 
ments. Clearly, evaluation as a part of the management support system-and the information it 
provides, whether positive or negative-can be crucial to managing the system. 

• Positive information resulting from an evaluation would suggest more of the same treatment 
(either continuing or increasing the project resources) is desirable. Negative information would 
suggest either adjusting, refocusing, or dropping the program. If vocational education evaluators 
present information in this context, they should stress that enlightened management demands 
data-based decision making! This implies, of course, that the client/decision maker will use evalu- 
ation results to choose among alternatives. . 



Increasing the Political Impact 
of .Evaluation Results 

The importance of evaluation results is, of course, based almost completely on utilization. 
However, equal treatment should be accorded all results within;the realm of their comparative 
importance. It is certainly permissible tcdraw attention to ."key" results, if other results are not 
omitted or hidden and if the importance of the key result is not made to appear more critical than 
is warranted. \ 
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If the evaluation results are politically important, so much the better; for the notion that 
politics and education do not mix is virtually prehistoric. There is no ethical problem in suggesting 
the political impact, so long as the validity and credibility of the evaluation are not compromised, 
and the objectives and the involvement of the evaluation clients have already been identified. 

Linking Results to Reduced Costs 

The notion of "the politics of less" has become an educational reality. Thus, the linking of 
evaluation results to measures of cost effectiveness/benefits will heighten interest anti improve the 
chance of utilization. Recommendations should be made with this fact In mind. 

Findings which lead to recommendations of high expenditures are virtually certain of rejection. 
It is often possible to highlight nonresource-intensive findings which can be implemented for little 
or no cost. WhiBn cost savings occur, especially if long-range cost-effectiveness or efficiency is likely, 
the potential for utilization is greatly enhanced! 



Broad Distribution of Findings ' 

Copies of the evaluation results {in various forms) should be distributed to.as many differjent 
audiences as possible. Remember, utilization need not be a one-time happening. Different clients 
can utilize different results at different times. Hence, the broader the distribution, the more utilization 
possible. 

; The evaluator should target different audiences (as previously mentioned) for evaluation report 
distribution, One large city Office of Research and Evaluation has identified a list of potential ^ 
audiences from which to determine dissemination strategies. This list appears as Checkiist 4. 

Writing a Report 

The virtue of brevity should be obvious, but it may be highlighted as follows: The briefer the 
document reporting results— 

• the less threatening it is,' , ^ 

• the more quickly it can be read, 

• the more comprehensive it js, 

• the less irrelevant material (filler) it contains, and ^ 

© the more direct (and apparent) are the recommendations. ' 

Decision makers tend to be busy. They want "short, sweet, and to the point" information. Those 
formats that require brevity and succinctness of style are fact sheets, abstracts, and summaries. 

Emphasizing Nontechnical Aspects of Findings. Rarely do primary clients demand technical 
detail. Though fechnical detail must always be available as back-up, it should not permeate the 
report. Typically, tech^nical detail should be included as an appendix or as a separate volume. The 
evaluation should incliljde only enough technical information to (1) convince clients that rigorous 
standards were followed, (2) enable clients to understand the measure(s) used and the relative/ 
comparative importance of-the results, and (3) convince clients that they are "technically" justified 
in making appropriate use of the information. A gooct substitute for technical information is simple 
graphics (bar graphs, pie charts, and so forth). Technical terminology in the body of the report should 
be avoided at all costs. It might be included in a separate, technical volume or appendix. 
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CHECKLIST 4 



Poteirtiai Audiences for Developing 
Dissemination Strategies 




/ Not 
Important 



Importance 

Somewhat 
Important 



Very 
Importaht 



Local Superintendent 


■ 1 

? • t \ 
t □ . 


□ 


□ 


SuperintendentVCabinet (top administration) 


□ 


□ 


□ 


District Superintendents 


□ 


D 


p 


Principals (ail or selected according to 






i, , 


district, project, personnel level) 






□ 


Parents' Council 


□ 


□ 




Advisory Committee(s)/Task Force(s) 


□ 


□ 


□ 


Program Directors 


ED 






Project Managers 


□ 


□ 


:□ 


Project Staff ^ 


r 1 
□ 


□ 


;□ 


Supervisors ^ 


□ 


f — 1 
□ 


1 1 1 


Board of Education r 


r 1 

□ 


□ 




State Department of Education 


1 ■ ~~| 


□ 


□ 


National Institute of Education (selected personnel); 


□ 


□ 


n 


U.S. Department of Education (selected personnel) 


I 1 


1 1- 

□ ; 


Lj 


Ofiice of Federal Programs 


1 1 

LJ 


□ 


□ 


Large City Directors of Research and Evaluation 


□ 


□ y 


□ 


Libraries, ' , 


LJ 


□ 


Q 


State Legislc\tors (selected, education committee. 








local delegation) 


□ 


□ 


■ □ 


City Council (selected or all education committees) 


□ 


□ 


□ 


Office of Information Services 




■ □ 




Home and School Council (officers, school 








representatives) 




□ - 


□ 


Local Business and Industry Groups 


□ 


, □ ■ 


■ □ 


Citizens Committee on Public Education 


□ 


' □ 


□ 


Current and Former Students 


■ □ 




n 


Parents' Union 


□ ■ 


• □ 


□ 


Other Community Groups 


□ 


□ 


□ 


Coliege/University Personnel (selected, regional) 


□ 


r D 


□ 


Other School Districts 


. □ 


□ 


□ 


Other Social Service Agencies (public, private) 


□ 


D 


□ 


Mayor's Office 




□ 


□ 


Public Media 


□ 


D 


□ 
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Keying Language to the Audience. Different audiences m^y often require different reports, 
but if this requirement's not feasible, the evaluator should select the lowest common denominator 
readily acceptable and understood by the largest client group or most important decision makers. 
The language should always be nontechnical, direct, and authoritative. Although the information 
may be of tremendous value and of enormous potential utility, if the language fails to communicate 
, it, it may never be utilized. 

Establishing Credibility through Review 

Three means of establishing the credibility of the evaluation findings are to (1) participate in 
outside reviews such as federal/state audits,.(2) present the findings at a national cohference such as 
AERA (American Educational Research Association) review sessions, and (3) publish the^ evaluation 
results in journals, or other periodicals. Each of these methods is a good vehicle, but each creates a 
time lag. For in-house program evaluators, these may not be feasible. 

The local evaluator may wish to consider a review by other selected experts, e.g., a university / 
professor or a school district research and evaluation director. This review should be imtiated early 
in the evaluatibn process, with final comments made shdrtly after public release of the report. Scho( 
district experts may be arranged for on a cooperative basis at little or no cost. Similarly, local univer- 
sity personnel may participate as a public service, or in exchange for cooperative research activities 
with the school district. 



Comparing Similar Studies 

The evaluator may consider initiating a search for sigiilar evaluations as early as possible. 
PossibliB types of comparative criteria include^methodology (approaches), outcomes, and utilization 
of results. Other studies with similar findings should be emphasized,'as should use of the findings. 

It may also be possible to link the results to traditional research studies and common-sense 
conclusions. The more reliable and predictable the results, the greater their perceivod validity and 
the more acceptable they will be to the evaluation clients. The more acceptable the results, the 
greater their potential for use. 



Assessing the Context , .l. -^-- - 

Failure to cpnsiderspeciaTsitL^^ disastrous. TFiough the situation may not be 

controllable, it should be anticipated and cited along with its possible effects. For example, if a 
school district suffered a teacher strike prior to tjie implementation of a program, the implementatic 
of the program would certainly be affected, as v\;ould its results. Similarly, any expectation of failure 
among clients may lead to siBlf-fulfilling prophecy. 

The provision of resources is necessary to implement a program according to objectives, as is 
the cooperation of project personnel in enabling implementadon of the evaluation. All such situ- 
ations such as the Hawthorne effect or the John Henry effect should also be examined. 
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Summary 

Many means of promoting the utilization of evaluation findings have been presented. It 
should be obvious, however, that each of these approaches requires planning. It is difficult to 
imagine utilization occurring coincidentally. To help ensure the utilization of evaluation findings, 
a proactive approach is essential. 

Given that the purpose of evaluation is to provide information for decision makers, it is 
incumbent upon the vocational education evaluators to make this information available.' If they 
fail to do so, then the question arises: Is the evaluation complete? 
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CHAPTER V 

PREPARING AND REPORTING EVALUATION RESULTS 



Background 

The evaluation components for most state and local education agencies include (1) program 
planning and operational processes, (2) student achievement^.and (3) student employment success, 
and (4) requirements for special populations. Procedures for collecting and processing the data 
have improved substantially since specif ic evaluation requirements were defined in the feder,al 
Vocational Education Act of 1963, especially with increased use of management information sys- 
tems (MIS). The computerized MiS in some state and local educatioaagencies provide numerous 
computer printouts revealing statistics on program enrollments, student achievement, and student 
employment success. However, some of this valuable information, once it has been processed and 
reported, is never used again. 

Evaluators, meanwhii^?, accuse decision makers of ignoring the data and information. Decision 
makers complain because the data and information are voluminous, next specific, and difficult to 
rerid and understand. It follows that evaluators need to assess their performance in the area of 
preparing and reporting evaluation results. 

This section provides a more detailed discussion on those factors considered to be critical for 
the preparation of the evaluation report. The section builds upon the ideas presented in Chapter I V. 

Specifically, this chapter describes types of evaluative data and presents specific recommenda- 
tions for preparing reports, packaging information, and disseminating evaluation findings in Order 
to increase utilization. The focus is on one evaluation component, student employment success. 
Emphasis is given, to those data relating to product evaluation, and specifically, the recycling 
decisions as defined in Chapter III. Checklist 5 asks readers to identify those strategies they employ 
for utilization purposes. 

Data Analysis 

Data analysis, commonly defined, is the tabulation,. organization; and summarization of the 
raw information collected during the evaluation. The analysis should relate data to the evaluation 
questions of the study, shape it into some usable form, and determine through the use of an 
appropriate statistical model or "practical/nonmathematical rhodel" if the findings are significant 
or due to chance variation. * " . ' 

pased on the evaluation goals as they relate td the decision-making structure and decision 
makers, plans should have been made to test specific hypotheses or answer specific questions, and 
collect the data needed. Fitting. quantitative data to a statistical model or fitting qualitative data 
to a nonstatistical model directs the evaluators' attention to certain aspects of the data aind suggests 
inferences which may be made. This model should be made explicit and its use should be justified 
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CHECKLISTS 

Strategies for Improving the 
Utilization of Evaluation Results 



Identifying key decision makers / 

■ /• 

Establishing utilization goals and/Objectives 

Creating a follow-up and follo^^^^through 
utilization plan / 

Providing technical assistance 



r 



Monitoring activities to affect use 



Ye 



□ 
□ 

□ 
□ 
□ 



No 

D 

.□ 

D 
D 
D 
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by its relationship to the decision-making structure, evaluation goals, and nature of data collected 
The analysis may be invalid if- 

1. relationships are not recognized in the choice of analytical methods models 
(quantitative, qualitative); 

2. data aggregation levels are not appropriate; 

3. assumptions for employing analytical methods or models are not satisfied. 



Data and Information Presentation 
It is important to note the distinction between data and information: 

Information comes from data, which are logical presentations of measure- 
ments, observations, and computations. Logical is here defined as orderly, • 
intelligible, objective, and capable of forming accurate relationships based 
on principles and rules of reason (Mercer and Koester 1978, p. 85). [italics 
added]. 

Not all data contain information capable of being assigned a useful meaning. 

Assigning meaning to data consists of taking the information resulting from the data analysis 
process and subjecting it to expert scrutiny. Such an examination should explain the meaning of 
the displayed information in terriis that are comprehensible to decision makers and other information 
users. These assigned meanings are useful for accountability, policymaking, and program improve- 
ment. 

For example, the interpretation of analyzed data on student employment success can best be 
performed by the professional staff, the evajlCjators, or data analyst. However, top managers in the 
organization at either the state or local level|5 Wiust feel comfortable with the fact that these individual 
organizations at either the state or local levels kre acquainted with vocational programs, are credible 
and congpetent evaluators familiar with probjerrts facing the managers, and have a deep concern for 
the target audiences they are serving. Equally^ impbc^ant to this process is the need for the evaluator 
or data analyst to view these data as integral elemeritsof a comprehensive management information 
system (MIS). , - 

•V '■ ' - ■ 

Moreover, these data need to be considered as only one element of information contained in a 
MIS. For example, a preliminary step in organizing the data interpretation and presentation function ' 
of the MIS is deciding what client format is appropriate. This could take the form of a cross-tabulation 
such as program area by function (i.e., personnel development, teacher inservice, curriculum develop- 
ment, and so forth), by a combination of areas and functions, or by any other grouping that facilFtates 
the needs of a specific education agency-state (SEA) orJocal (LEA). . 

State and local education agencies may present employment success in a variety of forms, 
depending upon the target users, such as the following: (1) the general public, (2) education planners, 
(3) educational administrators, (4) board of education members, and (5) program advisory committees. 

Each of these audiences has different needs. The general public does not usually require a 
^detailed report. A dne-to-three page report highlighting the results of the assessment of the former 
students' satisfaction with their training and job may be sufficient. Educational administrators and 
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the advisory council for vocational education may be interested only in a one-page executive 
summary and a list of conclusions and recommendations. In contrast, educational planners, 
classroom instructors, and placement personnel may need a detailed "technical" report to enable 
them to recommend or develop specific strategies for program changes. State administrators and 
planners may want to review final reports prepared for federal officials. 

The interpretation of the evaluation data in the oral and written reports is important. Careful 
attention is required to assure that the evaluation reports are not misleading. Many users will judge 
the entire study effort on the basis of the final report. If it is not clear and complete or does' not 
reflect all of the findings and limitations of the evaluation, the information may be ignored or mis- 
used.There is no guarantee that evaluation results will not be misused. However, full and open 
disclosure can help to assure that findings are not misunderstood. In sum, the evaluation report , 
should be sufficiently clear to the audience in its description of goals, procedures, and findings 
about what was done, why it was done, and what was learned. 



Reporting Recommendations 

A common mistake in reporting on the asisessment of former vocational students' employment 
success is to distribute the same report to all audiences. As discussed in Chapter IV, evaluators must 
identify the audiences whom the report is intended to serve and for what purpose-accountability, 
decision making, or program improvernent. It is not uncommon for an LEA or an SEA to distribute 
hundreds of final report copies of fifty or more paqes. This practice is not only costly but also of 
questionable value. The general public needs straightforward summary information, not technical 
"trivia" or sheer mass. The administrator who has neither the time nor the technical expertise to 
review the entire contents to recommend policy action or determine program decisions may never 
read it. Only the educational, planner, classroom/labpratory instructor, counselor/placement 
personnel may need to know all the^detailed information. Thus, an important consideration in the 
strategies for presentation and utilization is the length of the report. 1n a study to determine appro- 
priate reporting formats for educational decision makers, Brickell states: 

Top officials and management staff were more likely to ask for short report; 
program and project specialists were more likely to request medium or long 
report in their areas of specialization (p. 99). 

As a rule of thumb, short reports average one page, three minutes reading time; medium reports, ten 
pages, fifteen minutes; and loQg reports, one hundred pages, sixty minutes (Brickell 1974, p. 99). 

The alternative reports on student employment success must be weighed in terms of the 
targeted reading audience. At the state level, all three reports-short, medium, and Ipng-should be 
prepared. If the state conducts the assessment of student employment success t)f former students 
or a sample of the total population, it seems advisable that it also prepare statistical reports for 
each of the LEAs. At the local level, the detailed report and the executive summary should be 
minimum requirements.. 



Graphic Presentations 

Graphic presentations are an extremely useful and efficient medium for the presentation of 
quantitative data in a manner which facilitates the comparison of values, trends, and relationships. 
Graphic displays.have qualities and.values lacking in a, narrative presentation: 

. . 1 . They are more effective in creating interest and catching the attention of the reader. 



Z They provide visual relationships which are rrore clearly grasped and more easily ' 
remembered, 

3. They are more time efficient, since the essential meaning of large masses of 
j statistical data can be assimilated at a glance. 

4. They provide a more comprehensive picture of the problem, making for a more 
complete and better balanced understanding. 

5. They stimulate and facilitate analytical thinking and investigation which bring out 
hidden facts and relationships. 

Graphic presentations may take; ?ny of the following forms: 

1 . Rectilinear coordinate charts - The most frequently used of this type is the simple, 
arithmetic line chart. A number of useful variations exist; the cumulative curve chart; 
staircase curve chart, simple-surface or silhouette chart, staircase surface chart, 
multiple-surface or band chart, and the 100 percent surface chart. 

Z Bar and column charts - The major use of these forms is to facilitate the visual 
comparison of the magnitude. of coordinate items or parts of a total. The bars in a 
bar chart are arranged vertically in a column chart. Basically, there are at least eight 
types of bar ch&rts: the simple bar chart, the bar-and-symbol chart, the subdivided-bar 
chart, the subdivided 100 percent bar ^:hart, the grouped-bar chart, the paired-bar 
chart, the deviation-bar chart, and th& diding-bar chart. 

There are^also eight basic types of column charts: simple column charts, connected- 
column charts, grouped-column charts, subdivided-column charts, net-deviation 
column charts, gross-deviation column charts, float '^g-column charts, and range charts. 

3. Semilogarithmic or ratio charts - This type of chart i^ especially suitable for showing 
proportional and percentage relationships.' It is a good method for portraying' rates 
of change in a graphic way. This type of chart not only correctly represents relative 
changes, but also indicates absolute amounts uC the same time. The vertical axis is 
ruled logarithmically, and the horizontal laxis, arithmetically. The continued narrowing 
of the spacings of the scale divisions on the vertical axis is characteristic of logarithmic 
ruling. Op the other hand, the equal intervals on the horizontal axis are indicative of 
arithme. ruling. This chart is also sometimes referred to as a ratio chart because of 
the proportional relationships which it portrays. 

4. Frequency graphs and related charts ^ There are three basic types of simple frequency 
graphs: frequency polygon, histogram, and smoothed frequency curve. The cumulative- 
frequency graph or ogive is well suited to the following purposes: (a) to determine and 
show the number or proportion of cases above or below a given value, and (b) to 
compare two or more frequency distributions. 

Probability graphs are based on th.e normal frequency curve. This method provides a 
test for proportional asymmetry as well as demonstrating comparisons between 
empirical and theoretical distributions and prediction- 

5. Miscellaneous graphic forms - The pie chart may be used to show component 
relations. The various segments of a circle represent component parts of the total. 

' The trilinear chart simultaneously -"portrays three variables in the form of elements 
of a single function of activity. It is always a 100 percent chart, since the sum of the 
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three values indicated is equal to 100 percent. Trilinear charts are especially useful in 
portraying operating, productio.n, or other costs expressed by a threefold breakdown. 

The scatter diagram (scattergram) and other types of correlation charts show in 
graphic form the degree and typ^ of relationship or covariation betwsen two series 
of data. In statistical terms, the relationship between two or more variables is ' 
described as correlation. The fan chart portrays change for two different periods 
either by percentages or index numbers. As many as ten or fifteen items may be 
shown, depending on the range and scatter of values. 

Ranking or rating charts place emphasis on the position of certain items or categories. 
This position is usually based on magnitude or frequency. Therefore, emphasis is placed 
on rank-order position, rather than on the values themselves. 

The most effective way of showing spatial relationships is the map. Maps are often 
helpful in locating problems, testing hypotheses, analyzing data, and discovering hidden 
facts and relationships. The following basic types of maps may be useful in portraying 
statistical data: (a) cross hatched or shaded maps, (b) spot or'point-symbol maps, (c) 
, isoline maps, (d) maps with one or more types of graphs superimposed, and (e) a com- 
bination of two or more of the preceding types. 

Pictorial graphs and charts may be used with popular reports which are prepared for 
nontechnical use. They add interest for the reader who may not be otherwise. motivated 
to look at the item. Charts drawn in projection have limited use in vocational informa- 
tion/packaging, largely because of the complexity involved in designing three-dimensional 
pictorial graphic forms which are distortion-free. 

This is a very brief outline of some graphic forms which vocational education evaluators may 
consider for rej^orting evaluation results. The reader is referred to figures 3 through 9 for selected 
ideas for developing graphic displays. A more detailed description about each specific type with 
numerous examples may be. found by consulting the following references: 

AT&T. Making the Most of Charts: An ABC of Graphic Presentation, Washington, D .C. : . 
Navy Publications and Printing Service, iSlovember 1970. 

./ ■ 

Schmid, C,F., and Schmid, S,E. Handbook of Graphic Presentation, New York: John 
c Wiley and Sons, Inc., 1974. i 

Starr, H; et al. Selecting, Analyzing, and Displaying Planning Information, Columbus, OH: 
The National Center for Research in Vocational Education, The Ohio State Univer- 
sity, 1979. ^ 

A thorough discussion of technical strategies and techniques. to aid evaluators in communicatinn 
their evaluation results is found in the publication, A Communication Handbook for Researchers 
.and Evaluators by HoWey B\z\,{\Q7Q), - ^ — 



Preparing the Content and Information Packaging 

Information about vocational programs is generally prepared for one of three types of functions: 
(1) public information/relations, (2) administrative decision making, and (3) program decision making. 
Each requires a.different strategy for content development and style of packaging. Under normal 
circumstances, it is not advisable to photocopy the computer-generated printouts for distribution. 
The three types of functions for which information is prepared and packaged are treated separately 
in the following discussion. 
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FIGURES 

(Three-Dimensional Bar Chart) 
TOTAL £NI?OLLMENTS BY PROGRAM, 19-/19- 
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Vocational Program Code: 

^\ - \ ' 

1 Plastics"T6chnician 

2 HorticuTtMre Production 

3 Livestock (\/^anagement 

4 Fashion Retailing 

5 Insurance \ 

6 Marketing 

7 Accounting 



8 Quantity Food Preparation 

9 Auto Mechanics : 

10 Practical Nursing 

11 Data Processing (keypunch) 

12 Cosmetology 

13 Interior Design 
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FIGURE 4 • ' 

(UneGr^ph) 
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FIGURES 



(Bar Chart) 
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FIGURE 7 

(Column Chart) 
19 — GRADUATES BY VOCATIONAL PROGRAM AND SEX 



ACRI DISTRIB. HEALTH HOME 
ED. EC 



MALE 



FEMALE 



_i 



BUS. TECH. TRADE/ 
IND. 



43 



.51 



ERIC 



FIGURES 
(Bar Chart-Shading) 



AGRICULTURE 

DiSTRIB. ED 

HEALTH 

HOME EC. 

BUSINESS 

TECHNICAL 

TRADE/IND. 



POST HIGH SCHOOL ACTIVITIES 
OF THE19— GRADUATES 



25 



1 

1 




1 • hviV,Vi^ffl3«»MQaMM| 


r- ^ — 


r 




■J 


r 






1 







250 



750 



75 / 100 



(Overlapped Bar Chart) 



CosmtlQlogv 



60 

— r 



:.y::;!-:::vi::-:"v::i%%!.v^^^^ . 



3 EMPLOYED 



KSSSS NOT IN LABOR 
FORCE 



IN SCHOOL/ 
COLLEGE 

IN THE 
Ml LITARY 



3 UNEMPLOYED 



44 



FIQl/'RE 9 



. (Three-Dimensicnai iar Design) 
STUDENT ENROLLMENT BY PROGRAM BY COURSE 
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Public Information/ Relations 



This function is one of keeping the gerveral public Informed about the needs and achievements 
of vocational education. The information must be packaged in as simple and efficient a way as 
possible. Very elementary visual displays such as charts and graphs are highly recommended. 

W^^ether graphic or tabular techniques are used, three factors underlie satisfactory ^ 
. display of quantitative data: simplicity, clarity, and effectiveness. The graphic, 
and tabular forms of data dir^piay must be easily read and understood, and must 
be presented in a manner which will facilitate ease of comprehension and retention. 
These purposes require consideration of: (a) the nature of the data; (b) the purpose 
of ths display; (c) the medium for presenting the data; and (d) the audiences to ; 
vvhom the data are presented. One or all of these factors may be pertinent to any 
situation where data are presented or displayed (Starr pt al. 1979, p. 61 ), 

When the visual display «s complete, it should be examined critically in light of the following 
questions: » 

.1. Does it convey the intended message? 

2. Does it displav ^he relationship clearly? 

3. Can it be displayed differently? 

4. Does it have ey^ appeal? ' 

Although these are basic requirements, more often than not individuals who package information 
fail to address them and thus run the high, risk of failing to communicate effectively to the intended 
audience—the public. 



! Administrative Decision Making 

! . Contrary to popular belief, decision makers are generally not voracious report readers. Many 
\ simply do not have the time to do extensive reading and analysis. Because decision makers tend to . 
y be oriented more^toward oral communication^ it would obviously be a waste of time and effort to 
\ provide them with extensive sets of unsynthesized data, expecting them to do their own analysis of 
outcorries relating to pr6blems they are. trying to solve. Decision-making information offered to / 
managers should be synthesized and packaged intojj/ery practical, usable, "palatable" form. Charts 
, and tables of data need to be clear and concise for immediate understanding. ^ 

A great deal of responsibility rests on the evaluator or data analyst who must interpret and 
package the information for dissemination. , 

' " ■ , . / ' . • ■ . ' ' 

Program Decision Making 

Clients involved in program development at either the state or local level should recer/e the 
detailed report and the executive summary. The' raw data and processed information may also be ' 
prepared for individual institutions and agencies. 

Care must be taken to maintain confidentiality data ahd information. Schools should of 
course receive all information and data recorded about their own programs, as well as summary. data 
for the LEA andj region as a whole. In sharing such derailed data/information fromMndividual schools 
with other schools, consideration must be given to the federal legislation. requirements for the con- 
fidentiality and pjrotection of individual rights. ' ' ' , . 
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Factors to Consider in increasing , 
. the Use of Information . 

■/ • , 

The development of working relationships/both within the organization and among members 
of different organizations, is extremely important. One important internal working relationship is the 
linkmg of evaluation information with people involved in planning, programming, operations, and 
compliance. Here, the development of a conceptual framework is extremelyjmportaht. This involves 
the development of tactics, strategies, and procedures. ^ \ 

A fundamental question for school improvement is^ how does change take place? One can iDosit 
that it takes place in part through the use of existing systems. Another way is through a reordering 
. of priorities. 

V\lho are the key actors in this change or use procW Linkers and change agents ^re among 
those most commonly defined as such. Many linkers and change agents already exist in schools, e.g., 
teachers and curriculum coordinators. The problem is that chose individuals do not visualize their 
potential for the role. The challenge then is to adjust their focus on problems concerned with change 
^ or evaluation use-for example, allocating resources, developing communication skills, and imple- 
menting diagnostic procedures, c 

if , 

The importance of having clients participate in the design of an evaluation, use plan cannot be 
overstated. Meaningful collaboration-working within an existing network or de^jeloping new net- 
works—is essential. 

Effective linking of developi&rs, users, and evaluators begins with an awareness of what infor- 
mation systems do exist, either formally or informally. Serious consideration should also be given 
to using cost-effective communications technology, such as the electronic newsletter. 



l^r-Oriented Mode 

Another suggestion for increasing the use of evaluation data and information is to follow a 
user-oriented mode rather than a product-onfented mode. Agpin, plans should offer efficient service 
at a reasonable cos;. A number of strategies are available for the evaluator. With regard to efficiency 
in a people-oriented mode, the telephone as a medium for delivery can increase the probability of 
utilization. Another important strategy is to make initial contact and serve peopte with enthusiasm. 
Also, one must take into consideration the differences that exist among loQal school needs, state 
education agency needs, and individual needs. In all cases, the infprmation should be made available 
in a timely manner. The development of a network of coordinators can be a most cost-efficient 
method for transmitting evaluation data and information. The ombudsperson, as mentioned in the 
previous chapter, can serve as a key tg^this network concept. This reduces the number of direct 
contact points for the evaluator, and allows for the network to serve as a multiplier in dissemination. 
Other modes of communication which the evaluator can and should use are the news media, con- 
ferences, newsletters, and meetings v *th college and inservice classes. 

.J ' . ' i ■ 

Effective dissemination and use of evaluation results seldom, if ever, occur by accident. They 
must be planned. Hull and McCaslin (1977) offer thirty implementation techniques that represent 
tools which may be used to influence people toward implementing innovations. A number of those 
techniques can be considered by the vocational education evaluator for enhancing the use of evalu- 
ation results. For example, an evaluator should establish an evaluation utilization council composed 
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of teachers, program supervisors, teacher educators, parents, students, employers, and state education 
officials. This council can serve as a sounding board to review and advise on the packaging and 
dissemination of the evaluation results. There needs to be an effective delivery of usable information^ 
and a significant effort must be made by the evaluator in focusing on the attitudes of potential users' 
Checklist 6 presents ideas for organizing the evaluation council. 

Strategies and procedures for the use of evaluation must be targeted on the user groups. The 
evaluator needs to describe the potential use of evaluation results to individuals and groups. Specificall 
the decisidn setting(s) and decisibn type(s) of client groups such as students, teachers, administrators, 
and legislators must be analyzed individually or collectively. The analysis should focus on their use 
requirements in regard to program planning and operational processes, student achievement, and . 
student employment success. In sum, a number of factors or conditions exist that contribute to the'' 
utilization of evaluation results. But it is the preparation and reporting-as one might say, the "sum- 
mative activity"-which often determine whether or riot utilization occurs. Checklist 7 is presented foi 
readers tc rate these and other factors in order of importance. 



Scenarios of Evaluation Utilization 

There are innumerable variables assbciated with the decision making process which affect the 
complex nature of vocational education change. The utilization of evaluation results for effecting 
change is merely one segment of the total process of change, but should never be.underestimated as a 
critical component. 

The following scenarios depict instances where evaluation results were employed in the decision 
making process.^ln each scenario an evaluator is speaking. These examples were adapted frpm Weiss 
1977, Patton et al. 1978, Alkin, Daillak, and White 1979, and Franchak and Spirer 1978. 



Scenario 1 . 

Annual vocational education graduates' employer surveys in a large school district revealed that 
graduates lack essential employability skills. A closer look indicated that some^students did not know 
how to properly fill out an employment application, how to conduct themselves in a job interview, 
or how to keep a job once they found one. 

The vocational education director, other administrators, and occupational specialists met and 
formulated a plan to better prepare students for entering the job market. In several large high schools, 
separate courses on "employability skills" is now offered on the ninth- or tenth-grade level. Further- 
more, every high school, regardless of size, has instituted a unit of employability skills in certain 
academic and all vocational classes. These units are taught by either occupational specialists and 
vocational teachers, depending upon the situation. The program has met with favorable comments 
from employers and students, and recent survey data have tended to substantiate the changes. 



Scenario 2 

Employer surveys trom a small community college revealed that some graduates of^business 
programs were failing to meet the minimum typing performance standards required in certain occu- 
pations. The head of the blisiness department checked all typing programs and nbted that procedures 
for testing typing speed and accuracy were not consistent with those used in busfness and industry. 



CHECKLISTS 
Organizing the Evaluation Utilization Team* 



1. Organizing the Utilization^eam 

Other (jersons you know who might work in utilization roles with respect to this client 
system. 

Estimate of need 

* ■ for this type of 

. utilization role in this 
" Outsiders ^ Insiders ^evaluation effort 



As Catalysts 


Hi 


Med 


Lo 


As Process Helpers 


Hi 


Med 


Lo 


As Solution Givers 


Hi 


Med 


Lo 


As Resource Linkers 


Hi 


Med 


Lo 


Who are the members of the utilization team? 

j • Person 


Primary Skilfs 
or Contributions 
to Evaluation Effort 







nsiders 



Outsiders 



a. How good is the collaboration within the team? "^^ ^ 

b. Do you confer frequently? 

c. Do you share goals and values? 

d. Do you have a common plan? 

e. Do you divide up the labor rationally according \ 
to your evaluation skills? \ 



\ 



"Adapted from A Workbook of Checklists to Accompany tha Change Agent's Guide to innovation in Education by 
R. C. Havelock. Institute for Social Research, University of Michigan, January, 1973, pp. 3-4. 



\ 
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CHECKLIST? 

Factors and Conditions that Contribute 
to the Utilization of Evaluation Results 



Rating of Importance 

Very Somewhat Not too Not 
Factors and Conditions. Important Important Important Important 
r- • - At Al l 

1. Prespecification of goals and objectives 1 2 3 , 4 ' 

2. Identification of decision makers 1 ' 2 ' 3 4 

3. Understanding of the decision making - 

communication, and innovation processes 1 2 3 4 

Identification of informational require- ^ 
requirements ielevant to decisio . -j 2 3 4 

5. Evaluation report provides clear 

direction for future planning • 1 2 3 4 

6. Evaluation results support the decision 

makers' ideology 12 3 4 

7. Evaluation results suggest little or no 

change • 1 ,2 3 4 

8. Evaluator assumes active role in , 

promoting utilization .1 2 3 4 

9. Presence of a formalized evaluation 

system . 2 3 4 

10. Standardized tests matching objectives 1*2 3 4 

11. Use of reliable and valid instruments 1 2 3 4 

12. Evaluation results.which support an 
organization's continued existence 
(results not at variance with organi- 
zational goals) 1 2 3 ' 4 

13. Evaluators are internal to the 

organiza^tion 1 ^"^XZ^ 3 4 . 

14. Evaluators are external to the 

organization 1 2 ^ 3 4 

/ . * 

15. Involvement of evaluators early in 

development of program or project ,1 .2 3 4 

16. Place of evaluation capability in the 

organiz'jtional structure 1 2 3 4 

1 7. Responsibility fpr evaluation at a 

level appropriate to the decision — 

makers which evaluation is to assist 1 2 3 4 

18. Use of classical experimental research 

designs in evaluating programs 1 2 .3-4 
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Checklist 7 (continued) 



Factors and Conditions 



19. Developing evaluation reports 
according to the function, or 
role of the individual or groups 

20. Involving the program staff or 
persons being evaluated in the 
design of the evaluation study 

21. Exteiisive use of graphics in 

* development of the evaluation 
report 



Very 
Important 



Rating of Importance 



Somewhat 
Important 



Not too 
Important 



3 
3 



Not 
Important 
At All 



4 
4 



Don't 
Know 
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Because the word-per-minute rate^ were somewhat higher in the employers' tests than in the classroom 
tests, some students consequently failed to meet the minimum performance standards demanded in 
certain job situations. 

Consultations were held between the business staff members and individuals from government, - 
business, and industry to develop guidelines anci procedures^or teaching and testing typing skills to' 
match .those often required on. the job. Subsequent follow-up surveys have revealed a sharp drop in 
adverse comments pertaining to inadequate typing skills. \ . 



Scenario 3 

The evaluation study served two purposes. One was that it resolved a lot of doubts, confusions, 
and misunderstandings that our state advisory, comn^ittee for vocational education had and, second,' 
gave me additional knowledge to support facts that I already knew and, as I say, broadened the scope 
more than I realized. In other words, the perceptions of where the technical programs were going . 
and what they were accomplishing were a lot worse than ] had anticipated ... but I was somewhat 
startled to find out that they were worse, yet it wasn't very hard because it was partly confirming 
things that I was observing. 



Scenario 4 . 

We expected that the employer satisfaction with vocational training would be used, but in a 
vyay of providing background information around the consequences of certain kinds of state voca- 
tional education board decisidn-makin'g options, but not necessarily in and of itself determining those 
decisions. In other words, you might have some idea of what the consequences of the decisions are, 
but there might be a lot of other factors you'd take into account in how you would decide. 

The evaluation results had a particular impact in that they contributed to the general information 
context of what was going on at the time, rather than in itself . It contributes to that background of 
understanding the policy issues of vocational education meeting the heeds of employers, rather than 
resulting in one option versus another of policy being. - 



Summary 

In summary, interpretatiofi and packaging of data and information on student satisfaction with 
their training and job require strict attention to the needs and characteristics of the audiences for 
which the information will be prepared. Figure 10 identifies general factors to be observed in organiz- 
ing and formatting a report, and considerations for the graphic display of data. The reader is encouraged 
to review the summary checklist of strategies and techniques for improving the utilization of evaluation 
results. 
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FIGURE 10 

Factors to Consider in Preparing an Evaluation Report 



1. ■ Include thesurvey instruments in all re port) and presentations, if possible. 

2. Set up local reports based upon local requirements,, but also include information required by state and federal 
mandates that relates to local programs receiving state and federal support. 

3. Break down and analyze data as much as possible, but not all information and analyses need be presented 
to everyone. 

. ■■ ■ 

4: Make tabular summaries in the shortest possible form. Oral presentations are best vyith,lhis type of 

information. Follow survey form design when making presentation, question 1, 2, 13, and so forth. • ^ 

5. Prepare and present a copy of the report to each member of the audience during oral presentations, if 
practical. Be sure to present data in the form and content applicable to the particular audience. 

6. Summarized reports are usually the best format for presentation to most groups; however, more definitive 
information is requi; 2d in certain Instances. 

7. The comments section of a survey instrument is very Important. This area many times reveals needs and 
shortcomings, especially if a particular comment is repeated several times. 

8. Scan the comments section for-the most frequently mentioned items, and make summaries for inclusion in 
reports and presentations. Refrain from using actual names (teachers, administrators, etc.) given in comments. 

9. Do not attempt. to include survey information from' a student surveyed in the wrong curriculum area. For - - 
example, students are sometimes included in vocational surveys that may have only taken one vocational 
course, and are in fact pursuing college prep courses. Their inclusion in a vocational survey biaises the 
information and increases response error. 

10. Data tabulation and analysis methods must be appropriate, or hcnuse of the data is assured. 

11. Break out district data on a per school or program per campus basis. Most administrators are interested in 
information about their institution and their graduates. Compare schools to county data, or program to 
program data, but avoid institution to institution comparisons. 

12. Break out data by program, where applicable. This format is very meaningful for vocational reports and 
presentations. It gains teacher support for the activity. 

13. Compare and present data on courses and programs, not teachers. Many students react to questions about 
courses or programs by the nature of their relationship with a teacher, which biases the data. 

14. Kisep any sophisticated statistics in a separate section for those who wish to review them, but do not present 
statistics throughout a report. Rates of use will go down, especially in audiences with little or no background 
In research methodology. 

15. Percentages, graphs, and charts are information display methods to which most populations can best relate. . 

16. The main use of follow-up Information as it currently exists is to display general indications of the condition 
/ of an institution, district, and so forth. Include an abstract of each report that summarizes these findings 

and gives a brief description of the information gathering analysis.'techniques. 

17. Make sure reports appear neat, are printed on good quality paper; and are In readable form. 
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'CHECKLISTS 

Summary of Strategies and Techniques for 
Improving the Utilization of Evaluation Resujts 



Have you considered the need to: 


Yes 




1. 


identify the clients of the evaluation 


□ 


□ 


2. 


stress the importance or vitality of the program 


□ 




3. 


encourage active client participation in implementation of the evalua^tion 


□ 


□ 


4. 


guarantee the' technical purity of the evaluation 


□ 




5. 


use a variety of reporting and presentation formats 


□ 


□ 


6. 


pibn the sequence and determine the frequency of dissemination 


. □ 


□ 


7. 


key the results to the timing of decisions 


□ 


□ 


8. 


include an assessment of those objectives important to clients 




□ 


9. 


brief decision makers in advance of the release of the evaluation information 


□ 




10. 


deal with areas of resistance 


□ ■ 




11. 


use an 'ombudsperson 


□ 


□ 


12. 


adopt an entrepreneurial role 


. . □ ■ 




13. 


use an evaluation audit V 


□ 




14- 


establish the credentials of the evaluator 


□ 


□ 


''. y 


he aware of key power brokers in the program " ; * 


□ 


□ 




work with media representatives 


□ 




17. 


present the evaluation as an integral part of a management support system 




□ 


18. 


be concerned with the political impact of positive results 




■ □* - 


19. 


be concerned with the fiscal possibilities of the findings 

■ ■ ■ tf. 


□ 


r~i 


20. 


broadly distribute the findings 




□ 


21. 


be brief in presenting evaluation results 


□ 


□ 


22. 


emphasize the non-technical aspects of finding^ 


... □ 


□ 


23. 


key the language of the evaluation report to the audience 


□ 


□ 


24. 


establish the credibility of the findings by using an external reviewer 


□ 


' □ 


25. 


compare the. evaluation with similar reports 


□ 


□ 


26. 


be alert to special problems inherent in the agency or program.'s 


□ 


□ 




environment 










□ 
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APPENDIX 

V 'x ■■ . , • ' . V 

The following planning packet for a state program review follow-through is one example df a 
state's proactive approach twoard improving the use of evaluation results. The packet includes: \ ' 
(1) Guidelines for Developing Evaluation Results Follow-Through Plans, (2) Plan for Program Review 
Evaluation Results FoJIow-Through, (3) Worksheets for LEAs to Prepare Vocational Program\Evaiua- 
tion Results Review Follow-Through Plans. > \ 
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GUIDELINES FOR DEVELOPING 
EVALUATION RESULTS FOLLOW-THROUGH PLANS 

The results of the state-directed review of yocational programs will be transmitted in a summary 
report. The summary' report will present findings of the review team and of the state specialists, 
recommendations made by the review team, priorities identified by the state specialists. and 
suggested activities for appropriately responding to the recommendations. The local superintendent 
is responsible for developing utilization follow-through plans that are responsive to the findings and : 
recommendations of the program review. This guide is designed to provide suggested procedures for 
developing the follow-through plan. 

Written utilization follow-through plaris are to be developed in response to each recommendation 
. contained in the general section of the summary report and for each priority area and review item 
marked "ho" on the Individual Program Review Instrument. > 

The primary purpose of the vocational program review is to achieve program improvement. To 
accomplish this purpose, it is imperative that the variqus. individuals and groups involved in imple- 
menting the vocational program be involved in developing the utilization fol|ow-through plan for 
responding to the program review. It is strongly recommended that administrators, counselors, ^ 
teachers, advisory council members, and others be directly involved in developing plans for improving 
the vocational program and in formulating these plans into the utilization follow-through plan, 
following is a suggested procedure for involving the various groups and individuals in developing 
your utilization follow-through plan. \ 

Suggested Procedures for Developing 
Utilization Follow-Through Plan 

1. Superintendent disseminate copies" of the Program Review summary report to all administrators 
with vocational responsibility, all vocational teachers, all counselors, advisory council members, 

. vocational support personnel, and other decision makers. 

2. Superintendent and/or vocational administrator meet with vocational advisory council to 
obtain recommendations for long-range planning. 

3.. Vocational administrator meet with other administrators and vocational teachers or a committee 
of vocational teachers to develop initial responses to the general recommendations in the 
summary report. 

4. Vocational administrator disseminate initial responses to other administrators, teachers, 



Source: Research Coordinating Unit, Alabama State Department of Education, December, 1979. 
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counselors, and other persons involved in the planning process for their review and reactions 
to the initial response. y 

•5.. Vocational administrator synthesizes responses for various groups and individuals and develop 
revised utilization follow-through plans for general recommendations. 

■> . ' • 

6. Superintendent revievy revised plans. . 

7. Vocational administrator disserriinate the revised response to all individuals receiving copy of 
the summary report for responding ^o the priority items and the items checked "no" on the 
Instructional Program Review Instrument relating to their respective programs. 

. 8. Invite state specialists to assist teachers individually or in groups in developing plans for 

responding to the sumnriary report and to generate ideas for utilizing statejstaff and resources 
in implementing the utilization follow-through plans. This may be accomplished through an 
inservice meeting. 



9. Vocational administrator receivis initial responses from teachers individual 
synthesize into draft of total utilization follow-through plan. 



y or in groups and 



10. Superintendent review draft of utilization follow-through plan with vocational administrator 
and revise as necessary. 

11. Superintendent mail an approved copy of the final draft of the utilization follow-through plan 
^' to state Research Coordinating Unit. State Director and team representative review draft. 

12. Superintendent and appropriate administrators meet with -State Director of Vocational Educa- 
tion and team leaderfrom vocational program revievvto discuss utilization follow-through plan, 
negotiate items if necessary, and finalize the plan. 

13. . Superintendent receive multiple copiies of printed form of utilization follow-through plan. 

14. Superintendent and vocational administrator disseminate copies of the utilization follow-through 
plan to all persons recei<^ing copy Of the Program Review summary report. Disseminate the plan 

■ in workshop setting with instructions and training necessary for individuals to discharge their 
responsibility in implementing the total plan.' . 

15. Vocational administrator periodically monitor the progress being made in implementing all 
aspects of the utilization follow-through plan and report progress to superintendent. 

16. Superintendent and vocational administrator plan necessary action to fully implement utiliza- 
tion follow-through plan, utilizing state staff where needed. 



EKLC 



58 



65 



PLAN FOR PROGRAM REVIEW 
EVALUATION RESULTS FOLLOW-THROUGH 



Time Following 
On-site Review 

3 weeks 

4 weeks 



1 2 weeks 

1 2+ weeks 

13 weeks 

1 4 weeks 



1 5 weeks 



State Director mail one copy of summary report to local superintendent. ' 

RCU staff member deliver multiple copies of summary report and instructional 
Program Review Reference File to local superintendent. Staff present and 
discuss with, local administrator(s) Guidelines for Developing Utilization Follow- 
Through Plans (copy attached). 

Local superintendent mails a copy of the draft of the utilization follow-through 
plans to the RCU. 

Team representatives and State Director review draft df utilization follow- ' 
through plans. 

State Director and. team leader meet with local superintendent and other 
administratcrs to finalize utilization follow-through plan. RCU Director will 
substitute for State Director when State Director cannot attend. 

Team leader prepare typed copy of the utilization follow-through plan, print 
enough copies for local administrators, teachers, advisory council, board 
members, and others as appropriate. RCU distribute copJeS'^of local superintendent 
and to state staff. 

Appropriate state staff begin arranging to provide needed assistance to local 
staff for implementing utilization follow-through' plan. All state staff with field 
responsibility will review utilization follow-through plan to identify systems 
and programs needing assistance. Such systems and programs will receive 
priority in planning technical assistance. 

Appropriate state sta^f will monitor local programs to determine whether or not 
utilization follow-through plans are being successfully implemented. 

When successful implementation is not being achieved, state staff shall so indicate 
on their visit report. Visit reports which indicate that satisfactory progress is not 
being accomplished in implementing part of a utilization follow-through plan 
shall be routed through the Chief State Specialist tci the State Director. The State 
Director shall notify the local superintendent in writing of the need for remedial 
action. When satisfactory progress is still not achieved, the state superintendent 
will be advised. The state superintendent shall implement procedures to enforce 
state policies. (Technical Assistance Plan attached.) 



Source: Research Coordinating Unit. Alabama State Department of Education, December 1979. 
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WORKSHEET 1 



For LEAs to Prepare Vocaticna! Program Review 

Evaluation Results FoUow-Throuph Plan " . . 

INSTRUCTIONS: In column ^1 list the number for a specific recommendation in the summary report. !n column 2 indicate plai 
for implementing the recommendation. In column 3 indicate a suggested date at which time the recommendation could be 
implemented. In column 4 indicate any desired assistance from the state staff. " . 



/ 



Rec. 
No. 


Plans for Implementing 
. Recommendations 


Suggested Date 
for Implementing 
Recommendation / 


Requested 
Assistance 




1 ^ 

'm 


< 





lource: Research Coordinating Unit, Alabama State Department of Educiition, December 1979 
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WORKSHEET 2 

' For LEAs to Prepare Vocational Program Reliew 
Evaluation Results Follow-Through Plan \ 



INSTRUCTIONS: In column 1 identify a specific vocational in$tructional?program. In column 2 list the priorities, then the specific 
items which were checked "no" on the Instructional Program Review Instrument. In column 3 indicate plans for responding to the 
priority or for bringing that individual instructional program into compliance regarding the specific review item. In column 4 indicate 
the suggested target date for accomplishinajhe.plan.ln eolumn-5 indicatelnylssiSfan^ to carryout 

your plans. ' " I - - — . 



Program 


" frlority 
Items and 
Items 
Checked No 

( 


Plans for Responding 
1 to the Priority or 

BringingiProgram • 
into Compliance 


QimnpQtPrl n;)tp 

for Achieving 
„ Action 


Requested Assistance 
from State Staff 




\ 

9 




* 


; 7 






■' ' 1 




1 



Source: Research Coordinating Unit, Alabama State Department of Education, December, 1979 



GLOSSARY* 



Some of the words used in evaluation utilization elude^precise and clear definition. Those 
definitions that follow are intended to clarify the overall focus on evaluation utilization, but may 
be considered by some to go beyond the definitions needed in doing this type of activity. They are 
presented as a "definitional base" for which evaluation utilization can be understood and addressed. 
They are not "carved in stone" and should not be interpreted so as to foreclose alternative, 
potentially valuable definitions. 



Acceptance - A receptive or positive attitude toward an idea or judgment; an approving reception 
(English 1958), . hh y m 

Acceptance Behaviors ^ 

1. Exploratory Urge - An innovation tends to be more readily accepted if consumers perceive it 
as a discovery or as something to investigate. 

r . • 

2. Pleasure Principle - An innovation tends to be more readily accepted if consumers see the 
results of it as pleasant. Unpleasant activities tend to be discontinued. 

3. Group Influence - Consumers will tend to accept an innovation more readily if their peers have 
a positive attkiJde toward it. 

4. Reason - Consujners who are presentsd with, or who consider, both the positive and negative • 
aspects of an innovation ^re more likely to adopt that innovation than are the consumers who 
are presented with only one aspect (positive or negative). 

5. Stimulf^tion Drive — Consumers tend to accept jn innovation more readily if they see the input 
as exc ig. _ ^ 

6. Meaning Drive — An innovation will tend to be accepted more readily if consumers can clearly 
see the relationship between themselves and. the innovation. On a more concrete level, he may 
be wanting to know why an activity must be performed, i.e., how does tfie activity relate to 
something or someone else. 

7. Self-actualization — An innovation will tend to be accepted more readily if consumers perceive 
it as an opportunity to .grow emotionally, socially, intellectually, physically, or spiritually. 



\ 



8. Young of Age — An innovation will tend to be accepted more readily by consumers young in 
age than by older individuals. \ 



This glossary was developed prirnarNy from the publication, Assemblage of Change Process and Dif /us/on 
Terminologies (1973) by Lois Harrington, The Center for Vocational Education, The Ohio State University, 
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9. Cosmopolitan — An innovation will tend to be accepted more readily if consumers have a 
relatively wide range of experiences, i.e., if they have needed to cope with a diversity/df 
individuals, situations, environments, cultures, or subcultures. 

10. Personal or Organizational Affluence — Consumers with easy access to financiai^ar organiza- 
tional resources will tend to accept an innovation more readily than someone with few financial 
resources or with low status in the organization (EPIC Evaluation Center 197m. 

Accountability — A process applied to a program which parallels and is used in/j^junction with 
financial accounting. Expected outcomes of the learnmg experience are prestated in terms 
permitting pre- and posttesting to determine the extent to which objectives have been achieved, ^ 
and to permit a comparison of costs and benefits of various approaches to instruction. Stated 
objectives are expected to be realistic within legal, fiscal, and resource constraints, and to reflect 
current population needs for occupations prepairatibn, and current manpovver and job require- • 
. ments(AVA, March 1971). I . / 

Adaptability — Capacity of a school district to respond to change in o/der to meet the educational 
needs of the youth within the district (Harrell 1968). / 



Adaptation — Refers in the psychological sense to the process of est^lsbiftg-^nSlriaintainir^^ 
relatively stable, reciprocal relationship with the environment; for human beings this means the 
human, social, or interpersonal environment (Brody, in Maguire, Temkin, and Cummings 1971). 

Adoption — ' 

1. A decision to make full use of new ideas.as the best course of action available (Rogers and 
Shoemaker 1971) 

2. Process of institutionalizing a new program (Stuff lebeam 1&80). 

Adoption Process — The mental process through which an individual passes from first hearing about 
an innovation to final adoption (Rogers 1962). 

Anecdotal Evidence/- Casually observed incidents (English and Engljsh 1958). 

Anecdotal Record /- A record of casually observed events that seem to the reporter to have possible . 
significance (English and English 1958). ^ - 

Barrier — The static property of a system that stops messages, keeping them inside or outside of 
, the system (Havelock 1969). 

Change — 

1. An alteration in the structure of the organization, in any of its processes, or in its goals or 
purposes (Miles 1964). 

2. Quantitatively, a shift in size or scope of operations, acquisition of new skills, goals, values, 
or orientations. Qualitatively, a change may involve the substitution of parts or elements, an 
addition without changing old elements or patterns, restructuring, eliminating old behavior, or 
reinforcing old behavior (Havelock 1969). 

3. The process by which alteration occurs in the structure of a school system (Rogers, in 
Duncan 1972). 
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Change Agent — 



1. A professional who administers a procedure that is designed to produce behavioral, 
.^^j^titudinal, and personality change. Two essential criteria must be satisfied for the change 

agent: the individual must be acting in a professional rather than a personal capacity; and 
the impact of his actions must be directed toward producing change in others with whom he 
is working (Mann 1965). 

2, A professional who influences innovation decisions in a direction deemed desirable by a 
change agency (Rogers and Shoemaker 1971 ). 

** • . 

Change, Attitude — Three types: cognitive-oriented approaf^n — attempts to modify persons' 

attitudes by altering their beliefs about the attitude^^object through various forms of persuasive 
communications; affect-Oriented approach — both evaluations of, and behavior toward, 
particular attitude objects are modified by altering their emotion-arousing properties, usually 
through direct or vicarious conditioning procedures; behavior-oriented approach — getting e 
person to engage in new behavior in relation to the attitude object without untoward conse- 
quences (Bandura, Blanchard and Ritter, in Maguire, Temkin, and Cummings'1971). 

Change, Educationa! — 

1. Any change in schools or schooling that represents a perceptible departure from an existing 
norm ... to speak about a departure from the noi^m is to place no value on the direction or 
magnitude of the change (Smith 1972). 

2. A partial or whole alteration in p&ttern of an ongoing educational system (Harrell 1968). 

Change Process — A series of conditions, strategies, or procedures employed to effect a given 
alteration in a client system (Norton 1971 ). . 

Change Process, Educational — The methods and means by which an educational system or entity 
is altered in pattern or form (Harrell 1968). 

Change, Programmatic — That chiange which occurs according to a schedule or system under which 
action is taken toward a desired goal (Stuff lebeam 1970). 

Change, Social — The process by which alteration occurs in the structure and function of a social 
system (Rogers and Shoemaker 1971). 

Change, Technological — Ontologlcal view — invention and innovation are visible manifestations of 
a self-generating process or an institution having a dynamism and a life of its own; teleological 
view — invention and especially innovation are impersonal social processes determined by social 
or military needs or by the existence of an effective economic demand (Ayres, in Maguire 1971) 

Communication — . ^ 

1. Communication was conceived as an event linking pairs of staff members. Operationally, 
teachers and administrators were asked to specify others on staff with whom they talked 
regularly about school affairs and when two parties specifier une another, a communication 
bond was said to exist between them (Charters 1969). 

2. The process by which messages are transferred from a source to a receiver (Rogers and 
Shoemaksr 1971). \ 
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3. A high degree of dynamic dependence between two personal regions so that the changes 
in one bring about proportional changes in the other (Lewin, in English and English 1958). 

Communication Channel - The means by which the message gets from the source to the receiver. 

Communication, Formal - Information exchange between members of the organization through 
.formal channels provided by the organizational structure. 

Communication, Informal — Information exchange between members of the organization through 
channels other than those formal, specialized channels provided by the organizational structure. 

Communication Network — The set of channels employed by a group or individual when communi- 
cating with each other (Mann 1965). 

Community, Educational - Goes beyond the walls of the school buildings. It includes all of those 
groups that have a vital interest in the educational system: teachers, school administrators, 
students, board members, parents, and the community-at-large (Hitt). 

Compliance - A yielding to the desires, suggestions, or proposals of another person; has less 

implication of resistance or of. yielding unvyi llingly thain obedience (English and English 1958). 

Conflict - Involves an interaction between two or more individuals or alternatives as a result of 
difference in position or resource scarcity (Gerhardt and Miskell 1972). 

Conflict, Interpersonal — A relationship between two or more persons who seek goals that cannot 
be simultaneously attained under the prevailing conditions; a conflict within the individual 
that has root in his relations with others (English and English 1958). 

Confrontation — A situational setting where two or more cognitive factors come into contact with 
each other with the purpose of comparing or discerning likenesses and differences (Guerin, in 
Maguire, Temkin, and Cummings 1971). 

Consensus -r- A meeting of advocates and consumers who have the same primary goals and values. 
Such a meeting tends to result in the innova+'on being adopted at the integration level (EPIC 
. Evaluation Center 1970). ; - 

Credibility — The degree to which a communication source or channel is perceived as trustworthy 
and competent by the receiver (Rogers and Shoemaker 1971 ). 

Decision Makjng — The process of choosing from among two or more available options in response 
to some situation requiring altered action. I 

Decision Model, Synoptic Ideal - A change model aspiring to a thorough analysis of available 
solutions but requiring consideration of ali possible consequences for all possible! solutions in 
terms of all relevant success criteria; there is much information available to ] . concerning 
the probable success of your solution (EPIC Evaluation Center 1970). 

Decision Model, Disjointed Incremental - A change model aspiring to small changes and focused^ 
- mostly on current needs and problems; there is little information available to you concerning 
the probable success of your solution (EPIC Evaluation Center 1970). 
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Decision Model, Planned Change - A change mode! aspiring to large change (involving many steps) 
and focused mamly on long-range goals; there is little information available to you concernino 
the probable success of your solution (EPIC Evaluation Center 1970). 



Decision Setting, Homeostatic - Denotes restorative activity aimed ai the purpose of maintaining 
the normal balance in an educational system, and guided by technical standards and a routine 
cyclical data collection system (Stufflebeam 1970). 

Decision Setting. Incremental - A change setting in which your solution is gradually implemented 
•i for the purpose of continuously improving an education system; there is little information 

available to you concerning the probable success of your solution (EPIC Evaluation Center 1970) 

Decision Setting, Metamorphic - Denotes Utopian activity intended to produce complete changes in 
an educational system, based upon full knowledge of how to effect the desired changes (Stuffle- 
beam 1970). 

Decision Setting, Neo-Mobilistic A term contrived to convey the idea of change or movement 
toward the new-^enotes innovative activity for inventing, testing, and diffusing new solutions 
■ to significant problems-supported by little theorv or extant knowledge, yet the change is larqe 
(Stufflebeam ^970). - ~ — ; : 

Decision Situations, Homeostasis - Decisions to effect small changes supported by a high level of 
relevant understanding (Guba and Stufflebeam, in Maguire et al. 1971). 

Decision Situations,^ncrementalism.- Decisions to effect small changes suppor+'^^d by a low level of 
relevant understanding (Guba and Stufflebeam, in Ulaguire 1971 }. 

Decision Situations, Metamorphism - Decisions to effect large changes supported by a high level of 
relevanriinderstanding (Guba and Stufflebeam, in Maguire et al. 1971). 

Decision Situations, Neo-Mobilism - Decisions to effect large changes supported by a low level of 
relevant understanding (Guba and Stufflebeam, in Maguire et al. 1971). 

Decisions, Authority - Those forced upon an individual by someone in a superordinate pov/ver 
position. The individual has no Influence on the innovation decision (Rogers and Shoemaker 
1971). 

Decisions, Collective - Those which individuals in the social system agree to make by consensus 
- (Rogers and Shoemaker 1971). 

Decisions, Contingent Those which individuals in the social system agree to make by consensus 
(Rogers and Shoemaker 1S71 ). 

Decisions, Individual - The individua^ has some influence on the innovation decision (Rogers and 
Shoemcker 1971), 



Decisions, Optional - Made by an individual regardless of the decisions of other members of the 
system (Rogers and Shoemaker 1971). . ' . 



Diffusion — 



1. The (a) acceptance (b) over time (c) of some specific item— an idea or practice, (d) by 
individuals, groups, or other adopting units, linked (e) to specific channels of communication 
(f) to a social structure, and (g) to a given system of values or culture (Katz, Levin, and Hamil- 
ton 1963K 

2. (a) Placing an innovation with given characteristics into (b) a setting which has certain 
features and in which (c) the practitioner has customary ways of behaving (Brickell 1971). 

3\ The process by which new ideas are communicated to the members: of a social system, 
from a source to a receiver. Five stages: awareness, interest, evaluation, trial, adoption (Rogers 
19,5?/, Rogers and Shoemaker 1971). 

Diffusion Process — The spread of a new idea from its source of invention or creation to its ultimate 
usersusers or adopters (Rogers 1962). 

Diffusion Rate — The amount of t?me for an innovation or educational change to spread from a given 
locale to another (Harrell 1968). 

Diffusion Set — Selected persons in key positions within ihe system who have given full sanction to 
the change idea. The diffusion set functions to make the need for change that of other key 
persons within the system. This set attempts to get a definition of need from the client system ^ 
(Harrell 1968). 

Dissemination t- 

1. A controlled process of multimedjs communications through which information is passed 
to and gathered from targetiaudiencps in order to establish ievels of awareness and to induce 
reactions to an adoption of defined programs (Simmons 1968). 

2. A process similar to diffusion, but has three more ingredients: (a) the system within which 
the transmission takes place is well defined; (b) the transmission is a deliberate effort on the part 
of the source component; (c) the description and corjtrol of the transmission process vi/ithin the 
system is more or less complete and precise (Lin, Leu, Rogers, and Schwartz 1966). See Diffusion 

Distribution — Dissemination and distribution are two separate functions. The term distribution will 
be used H) refer to all of the functions associated with transferring, storing, insuring, mail'ng, 
handling, billing, packaging, collecting, replacing, etc. (Rosenau, Hutchins, Hemphill). 

Educational Process; — The entire activity that is directed toward formulating and ciccomplishing 
the objectives of the school system (Hitt). 



1. The process of delineating, obtaining, and providing useful information for judgingdecision 
alternatives (PDK, in Si jfflebeam, 1970). 

2. A systematic procedure whereby the quality of ihe teaching-learning process and thf; 
achievement of stated objejjt^es are ascertained. Ar ongoing process that provides input and 
feedback to guide change and offer directions for the program and its modification (AVA, 



Evaluation — 



\ 



lyiarch .1971). 
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Evaluation, Context - Defining the operating context, needs in the context, and problems underlying 
those needs (Stufflebeam, in Hock, Kean and Smith). 

Evaluation, Input — Assessment and identification of system capabilities, strategies; and designs for 
strategies to solve problems (Stufflebeam, in Hock, Kean, and Smith). \ 

Evfjkiation, Process - Identification of procedural defects in designs (Stuffiebeam, in Hock, Kean, 
and Smith). 

Evaluation, Product - Process of relating outcome information to objectives and context, input 
and process information (Stufflebeam, in Hock, Kean, and Smith). 

Facilitator — A decision, person, or condition that lowers the threshhold of resistance and aids the 
movement of an innovation toward a desired outcome (Hull 1972). 

Gatekeeper — A strategic role holder who is a middleman and monitors external information sources 
to a greater extent than his coworkers, so that he could relay to his coworkers useful knowledge 
from outside; one who stands guard over the entry points to the client system (Havelock 1969). 

Heuristic— . ; 

1. Serving to discover or reveal; notion of self discovery; problem solving with emphasis on 
imaginative and not routine problem solving. (Kerlinger 1973) ' 

2- Solution of a problem by a trial and error approach, frequently involving the act of learning, : 
and often leading to further discovery or conclusions without providing proof of the correctness 
of the outcome (U.S, General Accounting Office, October 1969). 

Information, Profession-Relevant — That information deemed desirable in achieving organizational 
objectives but not regarded as critical to the performance of the individual's routine fur:ction in 
that brgariization. 

Information, Work-Relevant (task oriented) — That information critical to the performance of 
routine functions of the organization's employees. 

Information System — Refers to a body of organized procedures for the collection, processing, snd 
dissemination of information— refers to oo.^n''inuinp systems and to systems established for one- 
time reports. The procedures may, but need not, involve the use of automatic data proi:essing 
equipment (Intergovernmental Task Force Information Systems 1968), 

Innovation — 

1. An idea, practice, or object perceived as new by an individual (Rogers and Shoemaker 1971). 

2. Innovation in education is the creative selection, organization, and utilization of human and 
material resources in new and unique ways which result in the attainment of a higher level of 
achievement of the defined goals and objectives (Purdy 1968). 

3. The introduction of a new element, idea or method 'nto a system (Harrell 1968). 

Innovation Assimilation ^ The extent of integration of an innovation or change into a defined social 
system or btitution (Lin, Leu, Rogers, and Schwartz 1966). 

\ , 
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Innovation, Technological - The process of translating knowledge into economic reality; involves 
four functions: (a) scientific (search for knowledge); (b) engineering {reduction to practice); 
(c) entrepreneurial (introduction to society); (d) managerial (optimization of usage). Each 
function requires a different type of skill and knowledge, involving some changes of attitudes 
and values, and requiring the manipulation of very different resources (Bright, in Maquire 
etaL1971). ^ . 

Innovator - The category of adopters who are first to adopt a new idea (Rogers 1962). 

Innovation-Decision Process - The mental process through which an individual passes from first 
knowledge of an innovation to a decision to adopt or reject, and to confirmation of this 
decision (Rogers and Shoemaker 1971). See Adoption Process 

Innovativeness - The degree to which an individual is relatively earlier in adopting new ideas than 
the other members of his social system (Rogers and Shoemaker 1971). 

Instructional Communicator - A primary change agent; school personnel knowledgeable in curric- 
ulum and instruction and public relations; generally, a person who is on the central office or 
supervisorystaff (Harrell 1968). 

Integration - The innovation becomes routine, i.e., is part of day-to-day working life of the teacher, 
or the administrator, or the user, whoever he may be (EPIC Evaluation Center 1970). 

Interest - Characterized by av':tive information seeking about the innovation. Although the consumer 
has an interest in. the innovation and a generally open attitude toward it, at this stage he has 
not made a judgment as to whether or not the innovation would be suitable for his own particular 
circumstances. As he gathers more information and' learns more about the innovation, the 
consumer's first positive or negative.attitudes toward it begin to emerge. Thesie feelings may 
prompt him to decide against adoption, or they may motivate him to move on to the next 
phase in the adoption process (EPIC^aluation Center 1970). \ 

Internalization — 

1. The change agent is perceived by the client as credible; the maintenance of any attitude 
change IS dependent on the relevance of the values to the issue (Kelman 1958). 

2. The extent to which a menriber of an organization perceives the innovation to be relevant 
and valuable to his role performance (Lin, Leu, Rogers, and Schwartz 1966). 

Leader - By definition, creates change (Lipham 1969). See Administrator 

Leadership - Interactive and situational; it arises only where a group follows an individual from 
free choice and not under command or coercion, and not in response to blind drives, but on 
positive or more or less rational grounds (Schmidt, in Bhola 1972). 

Legitimation - The approval or sanctioning of an innovation by those who informally represent 
the socigl system in its model norms and values and in the social power they possess (Rogers 
and Shoemaker 1971). 

/ ■ ■ i - ■ ■ 
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Model, Deterministic - A model in which. the variables take on only definite vaiues, that is, a model 
that does not permit any risk as to the magnitude of the variables; for example, a set of simul- 
taneous equations for which there is a unique solution. 

Model, Linear or Dependency - Views the goal of educational improvement as being dependent upon 
adequate diffusion mechanisms, which in turn require the invention and development of tested 
innovations to diffuse, which in turn depend upon the adequacy of the research base (Gideonse, 
in Maguire et al. 1971). 

Model, Linkage - Stresses the close interrelations of research, development, and dissemination . . . 
tend to be performer-onented and stress the importance of individuals in a research-development 
dissemination continuum (Gideonse, in Maguire et al. 1971). 

Model, Probjabillstlc - A model in which each variable may take on more than one value. Such 
models are sometimes called stochastic which means, literally, ".making a best guess" (U.S. 
General Accounting Office, October 1969). / • • 

Monitoring - To observe critically; to oversee, record or detect an operation, product or program, 
with instruments that have no effect on the operation, product, or program (Nikolai 1972). 

Need - The situation which occurs when what is actually happening is below that which is expected 
. {EPIC Evaluation Center 1970). 

Need, Felt — Something regarded as either desirable or necessary by the individual concerned, as 
distinct from something which, whether desirable or not, is not recognized as such by the 
individual in question (Good 1959). 

Norm — 

1. The most frequently occurring pattern of overt behavior for the members of a particular 
social system. The norms in a social system may be traditional and discourage the adoption 
of new ideas, or they may be modern and encourage the use of innovations (Rogers 1962). 

2. Standard or criterion for judgment (Good 1959). 

Observability — The degree to which the results of an innovation are visible to others (Rogers and 
Shoemaker 1971). 

Opinion Leaders - Those individuals from whom others seek information and advice (Rogers 1962). 

Opinion Leadership - The ability to informaliy influence other individuals' attitudes or behavior 
in a desired way with relative frequency (Rogers and Shoemaker 1971 ). 

Orientation-to-Change Score — An individual's degree of general predisposition toward change 
(Lin, Leu, Rogers, and Schwartz 1966). 

Power, Coercive - Based upon the expectation of punishment by those subject to power for failure 
to conform to' administrative goals (Warren 1969). 

Power, Expert — Derived from the extent of knowledge or perceived knowledge possessed by a ^ 
group or individual (Warren 1969). 
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Power, Legitimate - The superior need not to coerce or persuade subordinates in order to influence 
them, because they have accepted as legitimate the principle that some of their actions should 
be governed by his decisions (Blau, in Warren 1969). 

Power, Referent - A feeling of oneness ... or a desire for such an identity (French and Raven, in 
Warren 1969). 

Power, Reward - The client perceives that the change agent has the ability to administer rewards 
which are valued by the client, and that there is a reasonable possibility of receiving the reward. 

Practitioner - Ultimate adopter of an innovation (Brickell 1971). 

Primary Jnitiatorc - Key people in the system who act as change agents, spreading enthusiasm and 
excitement about a change idea. In educational change, primary initiators are generally key 
teachers. Administrative and supervisory personnel, excluding the instructional communicators, . 
are referred to as secondary initiiators. 'Initiators may be counted upon to give full. support and ' 
sanction to the change idea (Harrell 1968)^ „ _ ^„ ^ - 



Process Model - A general plan of action or design for bringing about a desired educational change. 
A model has specific devices and alternative strategies which vary from one locale to another 
(Harrell 1968). 

Product - A product is something produced by labor. It is a person or thing produced by or resulting 
from a process . . . books, monographs, and other publications; people, such as graduates; films; 
teacher training programs (Nikolai 1972). 

Product, Knowledge - Fills an importc^nt gap in our knowledge about subjects or topics relevant to 
education. The generation of that new information^shouid permit major progress to be made 
in either basic or applied activities; progress which would not have been possible without the 
creation of that new pro.duct ... a knowledge product may provide new information about 
effective learning strategies for elementary school children; or it may contribute new knowledge 
concerning more effective school management techniques; or it may provide data concerning 
the effectiveness of certain instructional programs ... the new knowledge product does not . / 
become a "product" until it is readily available to other educational practitioners. 

Products, Classes of — Information documents; training materials; installable systems. 

Program Developinent - Inventing, designing, and refining combinations of methods, materials, 
and subject content (Swanson 1967). 

Rejection — . 

1. A decision not to adopt an innovation (Rogers 1962). 

2. A direct response to planned change; reflects a lack of consensus between those initiating 
change and those implementing it (Sussman and Haug, 1 969, in IVIaguire et al. 1 971 ). 

Relative Advantage — The degree to which an innovation is perceived as better than the idea it 
supercided (Rogers and Shoemaker 1971). 
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Reliability - 

1. That part {of a result) which is due to permanent systematic effects, and therefore persists 
from sample to sample (Kendall and Buckland 1971). \ 

2. Accuracy: dependability; trustworthiness; the quality or qualities of a person or thing in \ 
virtue of which it can be counted on; the complex property of a series of observritions of a * 
measuring instrument, or of the entire measuring process, that makes possible the obtaining of 
similar results ».}pon repetition; tha degree to which such similar resuite may be predicted; the 
degree to which measurement is tree from random influehce (English and English 1958). 

Research — 

1. Process of creating new knowledge throuyii scientific method (Stuff lebeam 1970). 

2. Careful, critical, disciplined inquiry, varying in technique and method according to the 
nature and conditions of the prcbiem identified, directed toward the clarification or resolution 
(or both) of a problem (Good 1959). / 

Resistance — An ongoing mental decision to reject an idea for chang^ 

Risk — "Measurable uncertainty," per the economist Frank Knight?^ in decision theory, the 

distinction is made thatj:isk is measurable while uncertainly /s not. In situations of risk, the 
probabilities associated with potential outcomes are known/The term may be associated with 
situations of repeated events, each individually unpredictal&le, but with the average outcome 
highly predictable. In situations of uncertainty, the probabilities are not known (U.S. General 
Accounting Office, October 1969). .' I 

■ ' 1 ^ ■ 

Software — 

1. Telemedia: transmitted programs and mesjjges; recording media: recordied program 
materials (e.g., films, tapes, books, discy, etc) containing recorded messages; and both: working 
materials from which a program is created. May include scripts, written narration, audio or 
visual aids, etc., especially created or assem (Bretz 1971). 

2. The set of programs and routines used to operate a computer (Clubb and Traugott 1978). 
Strateyy — 

1. A gen9ral plan of action addressed to a broad, long-term objective . . . it consists of several 
specific tactics (Bricked 1971). / . 

2. Overall plan of action concocted in'the terms of the model (underlying theory) (Guba 1971). 

3. A unique set of mutually consistent diffusion techniques designed for difiusing a particuicr 
innovation to a particular clientele audience (Hull). 

Strategy, Integrative - A strategy for effecting chanqe in which "the change agent works WITH the 
change target, solving problems, educating and negotiatinq" (Weissman, in Fainstein and Fainstein 1972). 



Strategies, Advocate — 
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1. Coercion — the use of power by an advocate to force the installation of an educational 
change. This is.the most common-.y employed advocate strategy. !t is aiso the least effective 
in achievi.ig the integration level of adoption. In a coercive relationship, one of the parties, 
usually the advocate, has the ability to get the other party to try almost any new thing the 
powerful party recommends (EPIC Evalua/ioh Center 197G). 
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2. Confrontation — the head-to-head meeting of equally powerful and indeoendent 
advocates and consumers. In. thd confrontation strategy, the advocate and consumer have 
different primary goals, but are motivated to come together either by law or by other needs 
(EPIC Evaluation Center 1970). 



3. Coilaboration — the friendly meeting of advocates and consumers who openly seek 
cooperation from each other. In such a meeting, the consumer's goals usually "appear" to 
dominate those of the advocate. Appearances, however, may be created by the advocate to 
put the consumer at ease. In a collaborative relationship, the advocate is usually perceived as 
a consultant (EPIC.Evaluation Center 1970). \ 

A. Con:.<}fysus — the meeting of t^dvccates and consumers who have the same primary goals and 
values. Such a meeting tends to result in the innovation being adopted at the integration level 
(EPIC Evaluation Center 1970). 

System — Set of orderly and persisting interrelations between parts of a whole; all%8 elements 
that work toge;t^ier to perform a given function (English and English' 1958). 

System, Educatioria? — A collection of interdependent parts of a social system having definite. 
boundaries,/given to the accomplishment-Of educational goais (Harrell 1968). 

System, Open/- There is a dynamic interplay among the essential functional sLil:)pfocesses or 
subsystems in the org^nismic system which enables it to maintain itselfirva homeostatic 
steady state. Assuming a sufficient input of materials from its er>vrf6n.m8nf,vthe organism 
develops toward a characteristic state despite initial conditioris^(equifinajity),.>Ail bf this is 
accomplished through an automatic iislf -regulatory process (Hearn, in Miles, 1964), 

Systems Analysis — The process of breaking a system down into its constituentcomponent pailts, 
noting the relationships between the parts and between each part and the whole from wh ch 
it derives (Kaufman and Corrigan 1967). 

Trial — The individual applies the innovation on a small scale (Rogt^rs 1955). 

Triaiability — Tho degree to which an innovation may bt» experimented with on a limited basis\ 
(Rogers and Shoemaker 1971). 

Utilization, Knowledge — May be conceptu^^lized as a system and as a process. A system mode! of 
I utilization uses concepts such as ''ori^anization.!' group, person, agent, position, role, channel, 
and "link." A process , model inclu Jea such concepts as "relationship,'' linkage, transfer, 
exchange, translation, diffusion, arid communication" (Haveiock 1969). 

' ' i 

Validity- 

. J ■ ' 

1 . A test is valid to the degiee that we know what it measures or predicts. There are two 

basic approaches to validity: logJc^J BrrrJysis and empirical analysis. In loaical analysis, one ' / 
attempts to judge precisely what th;^ test measures. In empirical anu'.ysis, one attempts to show 
that the test is correlatea with soma otHf ; variable and therefore rtieasures the same thing 
(Cranbach 1949). • 

2. The quf-'lity of being grounded on truth or fact; the extent to which a test or other mec»suring 
instrument fulfills the purpose for which it is used; isually investigated by an analysis of test 
content or by a study of relationships between test So _.res and o^her vari-ibles (Good 1959). 

74 



81 



VaUdation — A procedure which i)rovides, by reference to independent sources, evidence that an 
inquiry is free from bias or otherwise conforms to its de^ared purpose; validity is to be 
contrasted with consistency, which i:oncefned wit]>the internal agreement of data or 
procedures among themselves (Kencall and Buckjand 1971). 

Vested Interest — An interest (as in ah existing political, economic, or social arrangement) in 

which the holder has a strong personal commitment (Webster 1965). . 

Volition — Act of deciding upon and initiating a course of action; action without external compulsion; 
voluntary activity (English and English 1958). 
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ANNOTATED BIBLIOGRAPHY 



Using Evaluation' Results 

Alkin, M. C; Daillak, R.; and White, P. Using Evaluations: Doss Evaluation Mal<e A Difference? 
Beverly Hills, CA: Sage Publications, 1979. 

The authors use the broader definition of evaluation utilization, challenge the noniitilization or 
underutilization of evaluation results reported in much of the literature. The results of five case 
studies of ESEA Title I and Title IV evaluations of school programs, with an.in-dept' fxamination 
of impact that the evalyation results have upon program decision making, are presented to support 
this alternative definition. A review of literature and a description of the case study method for the 
examination bf evaluation utilization is presented. Followed by the presentation of the five case 
studies, the final section of the book pr'esents a framework toward a theory of evaluation utilization. 



Braskamp, LA., and Brown, R. D., eds. "Utilization of Evaluation Information." New Directions for 
Program Evaluation, no.. 5. San Francisco: Jossey-Bass, Incorporated, J 980. 

The editor^ stiate that in editing this source book they have examined the many issues, concerns, 
- practices, and theories that need to be considered in studying and analyzing evaluation utilization. 
The book addresses four major questions related to evalustion utilization: (1 ) what is meant by 
utilization, (2) what is the role and function of the evaluator with respect to utilization, (3) what 
contextual and organizational factors influence utilization, and (4) how can evaluation results be 
communicated effectively? The editorj- i>tate that although there is no definitive solution to the 
problem of maximizing the usefulness of evaluation laforrriation, some important insights and 
suggestions on making evaluations more useful can be gained from those doing research and having 
experience in a variety of settings. " 



Braskamp, L A.; L^own, R. D.; and Newman, D. L. "The credibility of a Local Educational Program 
Evaluation Report: Author Source and Client Audience Characteristics." >4/77er/caA7 Educational 
Research Journal, 1 5, no. 3 ( 1 978) : 441-450. ^ 

Th^article describes the study of a series of simulations that have been conducted to examine the 
effects of source and message of educati6nal evaluation reports (evaluation of Arts Curriculum Pro- 
gram). The study focus was to determine if client, teacher, and administrators' responses to an 
evaluator's report (author identified as an evaluator, researcher, or art educator) are affected by the 
title and professional background of the evaluator, the part of the program being evaluated, and the 
client's organizational role status. The authors conceptualized the interpretation and use of program, 
evaluation as a process of several $tpps using a model that has guided research on communication and 
persuasion; vyho says what, how, to whom, and with wiiat effect. The study results suggest that 
source and audience characteristics influence client ratings of the evaluator but do not affec" changes 
ittagreement with the evaluator's recommendations. In summary, the authors state that little is 
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known yet about the effects of source, channel, message, nnd audience and their interactions on 
changes in client understanding of the program or product being evaluated, and on administrative 
actions of the clients within a decision-making environment. 



Cuba, Egon G. "Problems in Utilizing the Results of Evaluation." Jot/rna/ of Research and Develop- 
ment in Education, 8, no. 3 (1975): 42-45. 

This article focuses on the clients of the evaluation. The clients are defined as administrators and 
decision makers who fund, operate, or coordinate the educational or social action program being 
evaluated. The author identifies seven factors that affect the clients' utilization of evaluation results. 
These factors are (1 )/failure to satisfy the purposes of evaluation, (2) failure to meet the criteria of 
a good evaluation, (3) failure to^detect the discrepancies between program plans and program opera- 
tions, (4) failure to define the innate differences among the many audiences entitled to receive the 
evaluation information, (5) failure to identify the rapid pace of change, rapid fluctuations in the 
society itself, and frequent redirections in program mission that occur because of personal or policy 
alterations in the funding agencies, (6) failure to assess the integrity of the evaluator with regard to 
the programs they evaluate, and (7) failure to take into considercition the human and political factors 
that may affect the evaluation. In sum, Guba states, "the risk that evaluation information will not be 
readily utilizable is as a result of this conglomerate of factors, sufficiently high that the prud2n+ client 
will exercise great care in commissioning an evaluation, lest he find his investment to have been mis- 
spent." 

Mertens, D. M. "Methodological Considerations in the Use of Evaluation D^ta for Policy Making." 
.Paper pieser^ted at the 1980 Evaluation Research Society annual Meeting, Arlington, VA 
November 19, 1980. - \ ^ ' 

The paper focuses on the question, What type of information will Congress find useful in formulating 
policy on youth employment initiative, specifically, vocational education which affects high youth 
unemployment?" The author highlights a study of the effects of vocational education -for the purpose 
of providing information to policymakers (Mertens et al., 1980). This study attempted to assemble 
and summarize all studies that could be obtained on the^ffects of participating in vocational education 
that were reported from 1968 through 1979. Based on the analysis of those studies using predeter- 
mined standards for research quality, the author draws four conclusions which'affect evaluation 
utilization. One is that in terms of reporting research, more emphasis should be given to describing 
the educational program in order for researchers to know what the "treatment" actually is. Two, if 
sound conclusions are^to, be drawn concerning the effects of vocational education, sound research 
must be more pervasively conducted, reported, and disseminated. Third, researchers have an obligation 
to the educational community to report their findings in such .a manner jLhat they are accessible to 
others. Fourth, the author states that many authors have made recommend^^tions .o researchers on 
how to improve their responsiwness to policymai^ers' needs (Coleman, 1872; Florio et al., 1979; 
PincLs, 1980) while following s&ch recommendations has the potential of yielding better information 
for policymakers, it will not make the "real world" complexities go away. Finally, adding that harsh 
reality will forever limit the research methodologies and vyill result in conflicting findings. 
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Patton, M. Q. Utilization-focused Evaluation, Beverly Hills, CA: Sage Publications, 1978. 

ThjB author states that the contents of this book have emerged from many sources: studies of utiliza- 
tion; experience conducting evaluation; current theories of formal organizations and organizational 
dynamics; recent developments in decision making theory and policy analysis; work-in the diffusion 
of innovations and utilization fef knowledge; and many sources in the rapidly growing evaluation 
research literature. adds that the book records his experience in doing useful evaluation research 
from those who know. The book is considered both practical and theoretical. It presents a "how-to" 
on useful evaluative research and why to conduct it in the manner prescribed. He uses relevant 
literature and actual case examples very effectively to illustrate major points. 



Rutman, L. Planning Useful Evaluations: Evaluability Assessment Beverly Hills, CA: Sc<^ Publications, 
1980, 

The author establishes the rationale for the book based on his recogrw^on that many impact evalua- 
tions were ineffective because of the eyaluators' failure to determine v*ether tha program was 
structured and implemented in a manner that made it worthwhile tQ:svaluate; and if it wer^ feasible 
to implement methodologies to meet the study's purposes. Modifying and adapting the "EValuability 
Assessment" methodology developed by Wholey (1970) of the Urban Institute, Rutman provides 
direction for planning more useful program evaluations. 
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SERVING VOCATIONAL EDUCATION EVALUATION NEEDS 



A series of Evaluation Handbooks has been developed to 
respond to th^ need for evaluation of vocational education as 
identified in the Federal Vocational Education Amendments 
of 1976. Designed to provide in-depth assistance on a specific 
topic, they include — \ 



9 Evaluating Employer Satisfaction 
^^y^ ® Evaluating Student Satisfaction 
o Using Evaluation Results 



RD210 
RD 211 
RD 212 



$ 5.50 
$ 8.25 
$ 6.25 



• Guidelines and Practices for Follovi/-Up Studies of 
Former Vocational Education Students | 

© Guidelines and Practices for Foliow-Up Studies 
of Special Populations 

• The Case Study Method: Guidelines, Praf^tices, and 
Applications for Vocational Education 

• Performance Teiting: Issues Facing Vocational Education 

• EvssJuation Guidelines and Practices for State 
Advisory Councils 

/' 

o Specifications of Longitudinal Studies 



RD 171 

RD 172 

RD 189 
RD 190 

RD 188 
RD 191 



$13.00 
$14.25 

$ 5.50 
$11.00 

$ 3.25 
$ 6.75 



ORDERiNG hNJr^TRUCTIOPJS 



When ordering, please yse order number and title/' 
Orders of SIO.OO or less should be prepaid. Make 
remittance payable to the National Center >or 
Research in Vocational Educutjori. Mail order to: 

The National Center for Research 

in Vocational Education 
National Center Publicat^oni., Box F 
1960 Kenny Road 
Columbus, Ohio 43210 

Prices listed are in effect at th9 time of publication 
of this book. A. pfiCes include postage and handling. 
Prices are subject to change without notice. 



Quantity Discounts 

Orders of five (5) or more items, as listed by publi- 
cation order nurnber and tVlie', with j total dollar value 
for thf: orduT of :1 

$ 50 to S'ijOO, the discount is 5% 
S101 to $2p0. tht discount is ,10% 
$2G1 to $3p0. th2 discount i& 15% 
$301 to $4b0, the discCA^nt is 20% 
$401 and al)ove, the disco^/nt is 25%. ' 



Intern^tiorsaS Ort^ers 

All Orders, in any amount, from outside the United 
Stotes and. its possessions are to be paid in U.S. cu;- 
renry. Additional 'postage and handling'charges may 
be added for foreign shiprrnnts, if necessary. 



